how to make ZenTest autotest run whenever my tests change

rubyonrails-talk
1

Railsters:

Despite Rails being the only Web platform designed for TDD, a lot of its test infrastructure is still "cargo cult" - imitating other test rigs instead of understanding their principles.

Most importantly, tests should run instantly. There's no excuse for breaking this rule, and if you invent a platform that can't obey it then you are doing something wrong. And I have yet to meet a combination of editor and test script which does this. Even Visual Studio and C++ are faster and easier to work with.

The Rails problem is apparently rake db:test:prepare, which clones your database instead of detecting whether it needs a clone.

I have heard the fix is ZenTest's autotest. This allegedly loads the Ruby "VM" once, awaits file changes, and responds correctly to each one.

So I have a vanilla Rails 2.3.2 application, and I install autotest (and autotest -v does not work, so I can't see which version number I actually get), and the gem system claims I have version 4.1.3. And I run autotest in the project folder, and it says:

$ autotest (Not running features. To run features in autotest, set AUTOFEATURE=true.) /usr/bin/ruby1.8 -I.:lib:test -rubygems -e "%w[test/unit test/test_helper.rb].each { |f| require f }" | unit_diff -u Loaded suite -e Started

Finished in 0.000437 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

Note the system did not even run the whole test batch, first. And note it sees and warns that it rejects my features folder. I need it to see my test folder (NOT a spec folder!).

Then I make a non-trivial change to a correctly-named test file, save the file, and nothing triggers. (Pure Ubuntu Linux substrate, BTW.)

So how do I trigger?

If I could get that working, how can I TDD both story and unit tests AT THE SAME TIME, without using two different test runners?

2

sudo gem install autotest-rails. Autotest does not automatically intuit project directory structures, beyond the simplest, oldest Ruby idiom (lib/*.rb => test/test_*.rb). The autotest-* set of support gems adds support for more complex directory structures, like Rails'.

~ j.

3

John Barnette wrote:

0 tests, 0 assertions, 0 failures, 0 errors

sudo gem install autotest-rails

Yay thank you the good news is that runs now. (I would have gone with test/**/*suite.rb myself...)

The bad news is it's still slow as f---. Yes Gherkin is partly to blame. But does anyone have a replacement for rake db:test:prepare that does less??

4

Marnen Laibow-Koser wrote:

Despite Rails being the only Web platform designed for TDD,

Really? I have the impression that there are others...if nothing else, some of the Railsy PHP frameworks have copied this feature.

designed for - not copied on.

Most importantly, tests should run instantly.

In most cases, mine do. The exceptions are those which use regexps or DOM wizardry for parsing HTML. I suspect Oniguruma might help here, but I haven't tried it.

Rails??

RSpec + autotest + out-of-the-box test scripts + any editor you like. Done.

When the tests fail, what button do you type to navigate instantly to the test failure?

The Rails problem is apparently rake db:test:prepare, which clones your database instead of detecting whether it needs a clone.

When would it *not* need a clone? Tests are supposed to start from a clean environment.

We do not build a new CPU with new libraries and Ruby installation between each test run. All test isolation is a trade-off of some kind.

Jay Fields has posted some stuff on his blog about how to make Rails unit tests not touch the DB, but I don't really see much need to do that.

Thanks - that's the next thing I was going to ... rail at. Some consultants who teach TDD use a mind-trick to motivate their newbs to decouple their code. In a big-iron environment where database _migrations_ are expensive, until you pay that down, you can get a slight benefit to unit tests that reach into code that decouples from the database.

We should do that too. And when we need a fixture, to start such a test case, it should come from the database.

5

John Barnette wrote:

sudo gem install autotest-rails

Yay thank you the good news is that runs now. (I would have gone with test/**/*suite.rb myself...)

Excellent, glad it worked for you. Suites were originally bad for autotest because it attempts to do two-way mapping between test and implementation files, and suites generally aggregate a bunch of test files.

The bad news is it's still slow as f---. Yes Gherkin is partly to blame. But does anyone have a replacement for rake db:test:prepare that does less??

Are you absolutely certain that it's the DB load? Have you benchmarked 'rake db:test:prepare' against 'rake environment'? The latter is basically the minimum price you pay for a Rake task that loads Rails.

~ j.

6

RSpec + autotest + out-of-the-box test scripts + any editor you like. Done.

FWIW, this is still not competing with using Alt+Tab to switch to a console, then running script/cucumber features/whatever.feature. Autotest seems to just get mired in unit tests that I did not recently change, or something!

7

John Barnette wrote:

Are you absolutely certain that it's the DB load?

Nope!

> Have you benchmarked

'rake db:test:prepare' against 'rake environment'?

Of course not!

The latter is basically the minimum price you pay for a Rake task that loads Rails.

$ time rake db:test:prepare

real 0m2.153s user 0m1.324s sys 0m0.324s

$ time rake environment

real 0m1.623s user 0m1.316s sys 0m0.280s

Rhetorical question: So if that's the fast part, what is so f---ing slow???

8

Phlip wrote:

Marnen Laibow-Koser wrote:

Despite Rails being the only Web platform designed for TDD,

Really? I have the impression that there are others...if nothing else, some of the Railsy PHP frameworks have copied this feature.

designed for - not copied on.

Copying a feature in the design phase is still designing it in. Or were you going to say that Ruby wasn't really designed for most of its cool features because they were copied from Smalltalk and Lisp? :slight_smile:

Most importantly, tests should run instantly.

In most cases, mine do. The exceptions are those which use regexps or DOM wizardry for parsing HTML. I suspect Oniguruma might help here, but I haven't tried it.

Rails??

Huh? Oniguruma is what I haven't tried.

RSpec + autotest + out-of-the-box test scripts + any editor you like. Done.

When the tests fail, what button do you type to navigate instantly to the test failure?

Cmd-Tab. The test failure is usually where I was last editing. :slight_smile:

Seriously, you're asking the wrong person. I don't really care that much about that level of editor integration. If you do, I believe there is a TextMate bundle that will do what you want, but you will get better answers from others on this list.

[...]

We do not build a new CPU with new libraries and Ruby installation between each test run.

No, because we can guarantee that those bits won't be changed in a typical app. We cannot guarantee that about the DB, can we?

I *would* want a new Ruby installation for each test run if I thought my code were likely to modify the system Ruby installation as it ran.

All test isolation is a trade-off of some kind.

Sure.

Jay Fields has posted some stuff on his blog about how to make Rails unit tests not touch the DB, but I don't really see much need to do that.

Thanks - that's the next thing I was going to ... rail at. Some consultants who teach TDD use a mind-trick to motivate their newbs to decouple their code. In a big-iron environment where database _migrations_ are expensive, until you pay that down, you can get a slight benefit to unit tests that reach into code that decouples from the database.

We should do that too.

[...]

Do what? You mentioned several things, and I'm not sure which one you're referring to.

Best,