Improving Testing in rails and ruby as a probable GSOC project

[abhas_saroha]

hi all,
   a very rough idea has been developing in my mind as to improving
testing in general in ruby and rails to be a probable project.
        One of the ideas implemented to a certain extent and suggested
is by brynary to parallelize testing and using bonjour for auto
discovery of test servers.
  But i had a chat with him on irc and that in itself is not something
he believes requires work to last the duration of GSOC. So just
wanted some pointers from people about whether there has been requests
or a need from people to improve the test architecture that exists in
rails and ruby.
    If someone can give me direction to discussions people have had
regarding this it would be great help in expanding this idea. And
also do people think that there is a need to touch the testing modules
right now or is the way things work right now satisfactory for people
to give very little priority to this?

Abhas Saroha

[Chad_Woolley]

Shared test server farms are a lot of work and effort to get right. I
was just discussing this with someone yesterday. This is even more so
in the ruby world, where the environment can be really touchy (a wrong
gem version and everything can fall apart, unless you vendor your
entire world or lock down every dependency version, neither of which
are very practical - and this isn't even considering native dependency
versions such as imagemagick, browser versions, etc, etc).

Before going down this road, you should carefully weigh the costs of
dedicated test hardware (which is very fixed, especially with cloud
computing) against the engineering labor to maintain reliable shared
test servers, and all of the associated environment-management issues.
Are you really going to save money, effort, complexity, and time in
the long run?

-- Chad

[abhas_saroha]

Before going down this road, you should carefully weigh the costs of

dedicated test hardware (which is very fixed, especially with cloud

computing) against the engineering labor to maintain reliable shared

test servers, and all of the associated environment-management issues.

Are you really going to save money, effort, complexity, and time in

the long run?

the way i see it is that if a parallel testing implementation exists then implementing it for lets say a designated server farm or a cloud should be a similar thing. If you have multiple instances running on a cloud and they are able to identify themselves to each other through bonjour then isn’t it the same situation as local hardware of an organization being able to communicate and distribute a task? please correct me if i am oversimplifying things…

And if so wouldn’t it be good if an implementation exists in rails itself rather than each organization having to do it for its own needs?

Abhas Saroha

[Bryan_Helmkamp]

kaluza_klein wrote:

  But i had a chat with him on irc and that in itself is not something
he believes requires work to last the duration of GSOC. So just
wanted some pointers from people about whether there has been requests
or a need from people to improve the test architecture that exists in
rails and ruby.

To clarify, I said adding autodiscovery support to Testjour (which
currently lacks it since I rewrote it a couple months ago) was a few
day project, but there is plenty of work to be done on Testjour as a
distributed testing tool including:

* Fault tolerance
* Locking servers
* Supporting the cloud
* Supporting Test::Unit or RSpec (it's Cucumber only currently)

...and probably a lot more.

Cheers,

-Bryan

[abhas_saroha]

sorry for misunderstanding and misquoting you bryan.
Well this list does give me a lot to ponder over then.

Abhas Saroha.

[Ben_Brinckerhoff]

Abhas,

I just wanted to chime in on some of the interesting problems you
could potentially tackle in this space. We've been thinking about this
stuff quite a bit here at Devver, since we're building a commercial
solution to the same problem (we currently distribute Test::Unit tests
and are working on supporting RSpec). Although our solution is
currently
closed-source, we're excited about seeing an open-source alternative
since this is a big, challenging space and both solutions could really
help out the Ruby community.

Bryan - I haven't looked too closely at Testjour yet, so I'm frankly
not sure if you've already tackled any of these problems. If so,
please ignore them.

Test Detection - It's certainly possible to break up tests on a
file-by-file basis, but some performance improvements and interesting
features (see Test Ordering below) can be done if the tests are broken
up on a per-test level. However, this can be tricky to do. We're
currently experimenting with both approaches. Our per-test detection
can take a fair amount of time since it actually loads up the project
(which ultimately you have to do, since Ruby can dynamically generate
test methods). However, I suspect you could speed up the majority of
cases by using a parser and falling back to loading code (or executing
the entire file) when methods are dynamically generated.

Test Ordering - We're finding that for long-running test suites, just
distributing the tests can shorten the run time by 66-75%. However, it
seems that you could do better by figuring out the the optimal order
of running tests (e.g. run the longest-running tests first) and
determining which files are worth breaking up into individual tests
and which should be run as a file. Furthermore, you could make runs
seem faster by giving users the most useful information early. For
example, you could first run the tests that relate the code being
changed (like Autotest, but potentially more accurate), then tests
that tend to break often, then everything else.

Test Dependency Detection - In a perfect world, the tests in any test
suite should be able to run in any order, because they would all have
no side effects. Unfortunately, that tends to not be true for some
projects. It would be useful to have the system be able to
automatically detect which tests need to run before/after other tests
and keep this information around. This information would not only help
people more easily move their project to Testjour, it would also be
useful information about a way to improve their test suite.

Those are a few ideas that come to mind. I'd say that the features
Bryan mentioned are probably more important in the short term, but if
you're looking for difficult features to work on, there's plenty to
pick from

Let me know if you have any questions - I'm more than happy to help.

Ben

[abhas_saroha]

Thank you Ben for pointing out these points. I think these points can be taken account during the actual integration of testing modules with parallel architecture.
I will consult bryan about the direction of this project and see if these points can be included.
I have submitted a draft of my proposal and you are welcome to comment on it.
Abhas Saroha