Suggestion on Closing Tickets

I know that the issue of closing tickets was brought up a couple of
days ago and I'm hoping that others might be amniable to a suggestion
regarding how tickets should be closed.

To me the best way to prepare a ticket for closing is to actually
either a.) write a unit test which demonstrates that the ticket is not
applicable or b.) point to a unit test which does the same. To just
close a report with "works for me" or with "won't fix" seems to be
dangerous at best, especially when it is a bug report. By going
through the process of creating a unit test before closing we get the
benefit of proof, plus the framework gets better test coverage and the
test can provide the basis for additional tests in the future if more
information is provided for the issue.

Any thoughts on this? Does this seem like a reasonable thing to do?

V/r
Anthony Eden

I don't see how a test can be written to show something won't be
fixed. I also don't how a test can prove an issue doesn't exist.

If tickets are closed with 'works for me', the closer couldn't think
of a test that fails. It doesn't prove the issue doesn't exist, but
then, proving a negative isn't exactly easy. The onus should be on
those with an interest in reopening the ticket to provide a failing
test, rather than on those closing tickets to write passing tests.

Tom

Agreed - if you don't provide a failing test case with a defect, or at
least a very detailed bug report, don't be surprised if it gets closed
rather quickly. Thats how open source works - the burden is on the
reporter to prove the bug exists as best as possible.

- rob

I don’t see how a test can be written to show something won’t be
fixed.

“Wontfix” resolution is usually for requests for specific features be changed or enhanced. Real bugs should never be resolved wontfix because they are always fixed.

I also don’t how a test can prove an issue doesn’t exist.

Easy - take what the reported has described and turn it into a test. If it passes it is proof that it doesn’t exist, and you’ve raised test coverage a tiny bit.

If tickets are closed with ‘works for me’, the closer couldn’t think
of a test that fails. It doesn’t prove the issue doesn’t exist, but
then, proving a negative isn’t exactly easy. The onus should be on

those with an interest in reopening the ticket to provide a failing
test, rather than on those closing tickets to write passing tests.

I agree with this.

About defects that are reported with less than a minimal clue how to reproduce (and without a failing tests, obviously) - should they be resolved as “untested”, or that resolution only applies to ready patches?

Why can't we assume the reporter will come back to re-open the ticket if he or she feels the ticket should not have been closed?

Can you give an example of a non-trivial issue that has not been fixed just because someone closed the ticket for the wrong reason?

Kind regards,
Thijs van der Vossen

I don't see how a test can be written to show something won't be
fixed. I also don't how a test can prove an issue doesn't exist.

A passing test case which demonstrates that specific input results in
specific output indicates that an issue doesn't exist, at least using
the assumptions of the original issue. Granted a test case may pass in
one case and fail in another due to variations in the environment
(OSes, configurations, etc) however a test is better than no test. As
for test cases for something which won't be fixed, at least point to
or provide a test case which demonstrates the expected behavior. If
there is no test case then the correct behavior is a matter of
interpretation and will often lead to confusion, bugs, more people
raising issues, etc.

If tickets are closed with 'works for me', the closer couldn't think
of a test that fails. It doesn't prove the issue doesn't exist, but
then, proving a negative isn't exactly easy. The onus should be on
those with an interest in reopening the ticket to provide a failing
test, rather than on those closing tickets to write passing tests.

How do you know that the closer couldn't think of a test that fails?
Unless the closer creates a test that passes based on a set of
assumptions then how do you know what lead them to believe that it
works for them? We don't know what the closer's assumptions were, and
the point of test cases is to define something which can be applied
repeatedly with the same assumptions and which results in the same
outcome.

I think if an issue can be legitimately closed with a passing test
case then the onus does reasonably shift to the original person who
wrote the ticket to provide a test case that fails.

V/r
Anthony Eden

At least close it with untested if this is the case. Closing with
wontfix or worksforme is not the appropriate response, IMO.

V/r
Anthony Eden

Non-trivial is in the eye of the beholder, however here are some
examples of issues which have been closed in the last few days as
wontfix or worksforme, but which do not provide a test to verify.

http://dev.rubyonrails.org/ticket/2851
http://dev.rubyonrails.org/ticket/4415
http://dev.rubyonrails.org/ticket/6572
http://dev.rubyonrails.org/ticket/5455
http://dev.rubyonrails.org/ticket/5688

In all of these cases the closure is based on someone stating that
they do not see this behavior, or that they have tested it, but there
is no way to verify their tests. All I'm suggesting is that by
providing a test case anyone would be able to prove it on their own
environment, including the original author of the issue should they
still be experiencing the problem.

V/r
Anthony Eden

Anthony Eden wrote:

I don't see how a test can be written to show something won't be
fixed. I also don't how a test can prove an issue doesn't exist.

A passing test case which demonstrates that specific input results in
specific output indicates that an issue doesn't exist, at least using
the assumptions of the original issue. Granted a test case may pass in
one case and fail in another due to variations in the environment
(OSes, configurations, etc) however a test is better than no test. As
for test cases for something which won't be fixed, at least point to
or provide a test case which demonstrates the expected behavior. If
there is no test case then the correct behavior is a matter of
interpretation and will often lead to confusion, bugs, more people
raising issues, etc.

If tickets are closed with 'works for me', the closer couldn't think
of a test that fails. It doesn't prove the issue doesn't exist, but
then, proving a negative isn't exactly easy. The onus should be on
those with an interest in reopening the ticket to provide a failing
test, rather than on those closing tickets to write passing tests.

How do you know that the closer couldn't think of a test that fails?
Unless the closer creates a test that passes based on a set of
assumptions then how do you know what lead them to believe that it
works for them? We don't know what the closer's assumptions were, and
the point of test cases is to define something which can be applied
repeatedly with the same assumptions and which results in the same
outcome.

I think if an issue can be legitimately closed with a passing test
case then the onus does reasonably shift to the original person who
wrote the ticket to provide a test case that fails.

You are right to point out that variations in environment can cause a test to fail for the creator of a ticket, but succeed when tested by a Rails committer.

But the situation you are concerned about seems to be when the ticket creator hasn't given a failing (for him/her) test. You are proposing that the Rails committer who wants to close a ticket (i) interprets the reported problem in terms of a test, (ii) writes the test, and (iii) demonstrates that, in some 'standard' environment, the test passes.

In other words, you are proposing that Rails is presumed guilty until proved innocent, and asking the committers to act as the lawyers for the defendant.

The number of Rails users is in the tens or hundreds of thousands, with a wide variety of levels of skill and experience. You can see from reading the mailing list how often people raise 'problems' that turn out to be the result of their own error or misunderstanding.

The number of committers is small, and they are highly skilled. In the circumstances, the committers *must* be able to close tickets without spending a lot of time explaining why, and the burden must then fall on the ticket creator to justify reopening the ticket. Anything else doesn't make economic sense.

regards

   Justin Forder

I am actually more concerned with people who are not committers
closing tickets. One of the privileges of being a committer on a
project is that you have gained enough trust to warrent the right to
close tickets. Having said that, I still believe that *anyone* who
wants to close a ticket due to lack of tests should at least mark it
as untested and then close it. This is the common practice of the
Rails committers and is even documented in the needy report. By
marking it untested it appears in the needy report and can be followed
up on by others when time permits. Marking it as worksforme or wontfix
just seems wrong to me.

Anyhow, we can discuss this until we're blue in the face. At the end
of the day what matters is how we behave. To that end *I* will
continue to follow my own advice as to how to handle tickets for
Rails, and I encourage others to do the same, if for no other reason
than to contribute more test coverage to, and to inspire more
confidence in the stability of, the best darn web framework out there.

V/r
Anthony Eden

Anthony Eden wrote:

Anyhow, we can discuss this until we're blue in the face. At the end
of the day what matters is how we behave. To that end *I* will
continue to follow my own advice as to how to handle tickets for
Rails, and I encourage others to do the same, if for no other reason
than to contribute more test coverage to, and to inspire more
confidence in the stability of, the best darn web framework out there.

I think that in general, the onus is on the submitter to either
provide a failing test case, or simple steps to reproduce. If a
ticket gets closed prematurely, I figure there are a couple of
potential results:

0) it was invalid, everyone wins
1) The submitter reopens it and provides more information on
reproducibility, everyone wins
2) The submitter doesn't care any more and the bug stays closed.
everyone wins as we can go on to fix bugs which people care about.
3) The submitter doesn't care any more, but someone submits a new bug.
the bug's either better documented, or the problem repeats.

We don't have a dearth of users, and we're not penalising users who
submit invalid bugs. So I figure, err on the side of closing, reopen
with more information, not a bitchy comment. We're all after the same
thing here :slight_smile:

+1. If you've observed a bug and you really want it fixed, but can't provide a patch (for whatever reason), a failing test case is the best way to "prove" that it is a bug. That's not to say that every bug report must include a failing test case, but if you don't provide it, you shouldn't be angry if no one else can verify what you've reported. I believe this is a common philosophy in most OSS projects.

- Jamis

+1