Improving Testing in rails and ruby as a probable GSOC project

rubyonrails-core
1

hi all,    a very rough idea has been developing in my mind as to improving testing in general in ruby and rails to be a probable project.         One of the ideas implemented to a certain extent and suggested is by brynary to parallelize testing and using bonjour for auto discovery of test servers.   But i had a chat with him on irc and that in itself is not something he believes requires work to last the duration of GSOC. So just wanted some pointers from people about whether there has been requests or a need from people to improve the test architecture that exists in rails and ruby.     If someone can give me direction to discussions people have had regarding this it would be great help in expanding this idea. And also do people think that there is a need to touch the testing modules right now or is the way things work right now satisfactory for people to give very little priority to this?

Abhas Saroha

2

Shared test server farms are a lot of work and effort to get right. I was just discussing this with someone yesterday. This is even more so in the ruby world, where the environment can be really touchy (a wrong gem version and everything can fall apart, unless you vendor your entire world or lock down every dependency version, neither of which are very practical - and this isn't even considering native dependency versions such as imagemagick, browser versions, etc, etc).

Before going down this road, you should carefully weigh the costs of dedicated test hardware (which is very fixed, especially with cloud computing) against the engineering labor to maintain reliable shared test servers, and all of the associated environment-management issues. Are you really going to save money, effort, complexity, and time in the long run?

-- Chad

3

Before going down this road, you should carefully weigh the costs of

dedicated test hardware (which is very fixed, especially with cloud

computing) against the engineering labor to maintain reliable shared

test servers, and all of the associated environment-management issues.

Are you really going to save money, effort, complexity, and time in

the long run?

the way i see it is that if a parallel testing implementation exists then implementing it for lets say a designated server farm or a cloud should be a similar thing. If you have multiple instances running on a cloud and they are able to identify themselves to each other through bonjour then isn’t it the same situation as local hardware of an organization being able to communicate and distribute a task? please correct me if i am oversimplifying things…

And if so wouldn’t it be good if an implementation exists in rails itself rather than each organization having to do it for its own needs?

Abhas Saroha

4

kaluza_klein wrote:

  But i had a chat with him on irc and that in itself is not something he believes requires work to last the duration of GSOC. So just wanted some pointers from people about whether there has been requests or a need from people to improve the test architecture that exists in rails and ruby.

To clarify, I said adding autodiscovery support to Testjour (which currently lacks it since I rewrote it a couple months ago) was a few day project, but there is plenty of work to be done on Testjour as a distributed testing tool including:

* Fault tolerance * Locking servers * Supporting the cloud * Supporting Test::Unit or RSpec (it's Cucumber only currently)

...and probably a lot more.

Cheers,

-Bryan

5

sorry for misunderstanding and misquoting you bryan. Well this list does give me a lot to ponder over then.

Abhas Saroha.

6

Abhas,

I just wanted to chime in on some of the interesting problems you could potentially tackle in this space. We've been thinking about this stuff quite a bit here at Devver, since we're building a commercial solution to the same problem (we currently distribute Test::Unit tests and are working on supporting RSpec). Although our solution is currently closed-source, we're excited about seeing an open-source alternative since this is a big, challenging space and both solutions could really help out the Ruby community.

Bryan - I haven't looked too closely at Testjour yet, so I'm frankly not sure if you've already tackled any of these problems. If so, please ignore them.

Test Detection - It's certainly possible to break up tests on a file-by-file basis, but some performance improvements and interesting features (see Test Ordering below) can be done if the tests are broken up on a per-test level. However, this can be tricky to do. We're currently experimenting with both approaches. Our per-test detection can take a fair amount of time since it actually loads up the project (which ultimately you have to do, since Ruby can dynamically generate test methods). However, I suspect you could speed up the majority of cases by using a parser and falling back to loading code (or executing the entire file) when methods are dynamically generated.

Test Ordering - We're finding that for long-running test suites, just distributing the tests can shorten the run time by 66-75%. However, it seems that you could do better by figuring out the the optimal order of running tests (e.g. run the longest-running tests first) and determining which files are worth breaking up into individual tests and which should be run as a file. Furthermore, you could make runs seem faster by giving users the most useful information early. For example, you could first run the tests that relate the code being changed (like Autotest, but potentially more accurate), then tests that tend to break often, then everything else.

Test Dependency Detection - In a perfect world, the tests in any test suite should be able to run in any order, because they would all have no side effects. Unfortunately, that tends to not be true for some projects. It would be useful to have the system be able to automatically detect which tests need to run before/after other tests and keep this information around. This information would not only help people more easily move their project to Testjour, it would also be useful information about a way to improve their test suite.

Those are a few ideas that come to mind. I'd say that the features Bryan mentioned are probably more important in the short term, but if you're looking for difficult features to work on, there's plenty to pick from :slight_smile:

Let me know if you have any questions - I'm more than happy to help.

Ben

7

Thank you Ben for pointing out these points. I think these points can be taken account during the actual integration of testing modules with parallel architecture. I will consult bryan about the direction of this project and see if these points can be included. I have submitted a draft of my proposal and you are welcome to comment on it. Abhas Saroha