I just wanted to chime in on some of the interesting problems you
could potentially tackle in this space. We've been thinking about this
stuff quite a bit here at Devver, since we're building a commercial
solution to the same problem (we currently distribute Test::Unit tests
and are working on supporting RSpec). Although our solution is
closed-source, we're excited about seeing an open-source alternative
since this is a big, challenging space and both solutions could really
help out the Ruby community.
Bryan - I haven't looked too closely at Testjour yet, so I'm frankly
not sure if you've already tackled any of these problems. If so,
please ignore them.
Test Detection - It's certainly possible to break up tests on a
file-by-file basis, but some performance improvements and interesting
features (see Test Ordering below) can be done if the tests are broken
up on a per-test level. However, this can be tricky to do. We're
currently experimenting with both approaches. Our per-test detection
can take a fair amount of time since it actually loads up the project
(which ultimately you have to do, since Ruby can dynamically generate
test methods). However, I suspect you could speed up the majority of
cases by using a parser and falling back to loading code (or executing
the entire file) when methods are dynamically generated.
Test Ordering - We're finding that for long-running test suites, just
distributing the tests can shorten the run time by 66-75%. However, it
seems that you could do better by figuring out the the optimal order
of running tests (e.g. run the longest-running tests first) and
determining which files are worth breaking up into individual tests
and which should be run as a file. Furthermore, you could make runs
seem faster by giving users the most useful information early. For
example, you could first run the tests that relate the code being
changed (like Autotest, but potentially more accurate), then tests
that tend to break often, then everything else.
Test Dependency Detection - In a perfect world, the tests in any test
suite should be able to run in any order, because they would all have
no side effects. Unfortunately, that tends to not be true for some
projects. It would be useful to have the system be able to
automatically detect which tests need to run before/after other tests
and keep this information around. This information would not only help
people more easily move their project to Testjour, it would also be
useful information about a way to improve their test suite.
Those are a few ideas that come to mind. I'd say that the features
Bryan mentioned are probably more important in the short term, but if
you're looking for difficult features to work on, there's plenty to
Let me know if you have any questions - I'm more than happy to help.