How can I have the tests for my Rails app to be executed in a random order?

Hi How can I have the tests for my Rails app to be executed in a random order? Is there a simple solution using rake?

I already posted my question to stackoverflow (http:// stackoverflow.com/questions/1376267/ruby-executing-tests-in-a-random- order-with-rake) but the few responses I got were not able to solve the problem.

Thanks a lot!

Pietro Di Bello

Should, but it is possible for this not to be true. A random order would be hard to work with though - Run your suite and a test fails, run it again and it doesn't any more.

Fred

Marnen Laibow-Koser wrote:

Piero Di Bello wrote:

Hi How can I have the tests for my Rails app to be executed in a random order?

At least with RSpec, specs are executed in unpredictable order. I think it's the same each time, but it's not the order in which they appear in the spec file.

But why do you need this anyway? Each test should start from a clean slate, so the order should be completely immaterial.

First I'll say I completely agree with this. If tests/specs fail or pass depending on the order of execution then there must be serious problems with your test suite.

Now to address the OP. I know of nothing "built-in" any of the testing frameworks for this. What I suppose you would need to do is write your own script that searches though your project for all test classes then executes them in a random order.

This, however, would only randomize the testing classes, each test within each class would still execute in whatever order the test framework decided to run them. Which, would mean that you would probably also have to replace the test runner with your own that could randomize each method of each test class.

In conclusion if this is a solid requirement of your project, I'm really glad it's your project and not mine. :slight_smile:

Good luck.

Piero Di Bello wrote: > Hi > How can I have the tests for my Rails app to be executed in a random > order?

At least with RSpec, specs are executed in unpredictable order. I think it's the same each time, but it's not the order in which they appear in the spec file.

Unfortunately we don't use Rspec on our project, although I think we may give it a try

But why do you need this anyway? Each test should start from a clean slate, so the order should be completely immaterial.

Well, actually, the answer is in your words :slight_smile: Each test *should* start from a clean state, but what if it doesn't? For us it's a great value to have a suite of test executed every time in a random order (better, every single test method in every test class should be "shuffled"). Dependency between tests is a really bad anti-pattern, and we want to know if this happens as soon as possible.

anyway, thanks for your response, Piero

Marnen Laibow-Koser wrote: > Piero Di Bello wrote: >> Hi >> How can I have the tests for my Rails app to be executed in a random >> order?

> At least with RSpec, specs are executed in unpredictable order. I think > it's the same each time, but it's not the order in which they appear in > the spec file.

> But why do you need this anyway? Each test should start from a clean > slate, so the order should be completely immaterial.

First I'll say I completely agree with this. If tests/specs fail or pass depending on the order of execution then there must be serious problems with your test suite.

Now to address the OP. I know of nothing "built-in" any of the testing frameworks for this. What I suppose you would need to do is write your own script that searches though your project for all test classes then executes them in a random order.

This, however, would only randomize the testing classes, each test within each class would still execute in whatever order the test framework decided to run them. Which, would mean that you would probably also have to replace the test runner with your own that could randomize each method of each test class.

In conclusion if this is a solid requirement of your project, I'm really glad it's your project and not mine. :slight_smile:

Thanks for your ideas. I'll definitely explore some of them. No, it's not a solid requirement, but as I already said in a previous answer, we find it *very* useful to discover dependencies between tests as soon as possible (and probably the Rspec authors' find this important too, given that Rspec execute specs in a random order...).

Thanks Piero

The whole point of randomizing test execution is to have a better chance to discover that the tests are not independent. Once you find a test that fails because of the order of execution, you fix it so that it does not happen anymore.

Matteo

> Should, but it is possible for this not to be true. A random order
> would be hard to work with though - Run your suite and a test fails,
> run it again and it doesn't any more.

The whole point of randomizing test execution is to have a better chance to discover that the tests are not independent. Once you find a test that fails because of the order of execution, you fix it so that it does not happen anymore.

I get that, but unless you could say 'run the tests again but in the same random order as before' how are you going to be sure that you've actually fixed your code/tests ?

Fred