Why test Rails, itself?

rubyonrails-talk
1

I'm trying to learn the habit of writing tests, but now that I'm getting into it, I'm starting to realize that many potential tests -- including some described in Agile Web Development w/Rails -- don't test the application. Instead, they test Rails itself.

For example, what is the point of writing a unit test that simply tests a validate statement in the model? Yes, it's interesting (the first time, at least) that the test works and that, golly-gee, Rails works as advertised, as well, but is there any real use in doing this?

I can imagine a scenario where a I update Rails and suddenly a unit test that tests a Rails validator fails, but I expect the Rails development team would find this before me. Or is that too naive?

The reason I ask is that I'm starting to realize how time consuming it is to write tests... it could easily double the amount of work. I can't afford that, so am trying to focus tests on my application code. I just want to make sure I'm not missing some obvious benefit of testing the whole Framework.

-- John

2

Hi --

I'm trying to learn the habit of writing tests, but now that I'm getting into it, I'm starting to realize that many potential tests -- including some described in Agile Web Development w/Rails -- don't test the application. Instead, they test Rails itself.

For example, what is the point of writing a unit test that simply tests a validate statement in the model? Yes, it's interesting (the first time, at least) that the test works and that, golly-gee, Rails works as advertised, as well, but is there any real use in doing this?

I can imagine a scenario where a I update Rails and suddenly a unit test that tests a Rails validator fails, but I expect the Rails development team would find this before me. Or is that too naive?

The reason I ask is that I'm starting to realize how time consuming it is to write tests... it could easily double the amount of work. I can't afford that, so am trying to focus tests on my application code. I just want to make sure I'm not missing some obvious benefit of testing the whole Framework.

It's a very interesting question. There really is no point testing validates_format_of itself, since there's already a test suite for it. I'd say, though, that it's good to have tests that walk your models through everything they're supposed to do. Among other things that will come in handy if you accidentally comment out or alter the validation. So you can think of your test suite as a kind of documentation or mega-assertion pertaining to your whole application.

David

3

I would add to that:

1) If you do not test it, you do not know if it works. Yes there are tests of Rails itself, but not of your use of Rails. 2) The comment I have seen and come to believe is that if it is not tested then it is not guaranteed to work. The tests define what "work" is.

On a small project the programmer may know all the requirements and remember them at all times, and always get it right. :slight_smile:

But, even on a 2 man project it helps to build tests for what you expect the software to do.

The application tests will only test the cases that matter to the applicaiton, while the Rails tests test all the cases the framwork supports. But, you never know if the Rails tests cover the one case you care about.

Michael

4

For example, what is the point of writing a unit test that simply tests a validate statement in the model? Yes, it's interesting (the first time, at least) that the test works and that, golly-gee, Rails works as advertised, as well, but is there any real use in doing this?

+1

The reason I ask is that I'm starting to realize how time consuming it is to write tests... it could easily double the amount of work. I can't afford that, so am trying to focus tests on my application code. I just want to make sure I'm not missing some obvious benefit of testing the whole Framework.

+1. Actually, *it* doubles the amount of work. Imagine you want to test you method Father.add_kid. You'll need to 'manually' add a kid to your @father and then call your method add_kid and assert equality. You simply rewrite your code! That really annoys me.

However, testing is *really* useful, but not for testing if rake test:units passes: I have discovered 'bugs' (logic problems) when I was writing tests. You know, you're beginning to write your test_father_add_kid method with weird @father and suddently a flash[:warning] rings in your head: 'Hey! What about if @father has already 12 kids? It can't work because... blablabla'. So you correct your 'real method' and cut/paste in your test just for 'fun' and be proud to see that it passes.

My $0.02.

5

Hi --

I would add to that:

1) If you do not test it, you do not know if it works. Yes there are tests of Rails itself, but not of your use of Rails.

Yes -- that's what I mean about not testing validates_format_of itself, but testing what you want your application to do, and if what you want it to do involves calling validates_format_of, then the test will fail if v_f_of is broken.

2) The comment I have seen and come to believe is that if it is not tested then it is not guaranteed to work. The tests define what "work" is.

That still leaves the question: at what point can we consider x to have been tested?

On a small project the programmer may know all the requirements and remember them at all times, and always get it right. :slight_smile:

But, even on a 2 man project it helps to build tests for what you expect the software to do.

I think testing is good even if you're working alone on a project. But that's for another discussion :slight_smile:

The application tests will only test the cases that matter to the applicaiton, while the Rails tests test all the cases the framwork supports. But, you never know if the Rails tests cover the one case you care about.

I think you can usually assume it doesn't. But it's still a question of what point you hand off the responsibility to the framework -- or the language. I don't have any code that specifically tests String#capitalize, even though I use it and Rails uses it, because I assume that there are tests in place that test it, in both the Ruby and the Rails test suites. But if I have a method called title_case, or something, I'll test *that*, and if it fails then I can investigate where it went wrong.

All of this relates, also, to the use of tests as an anchor for refactoring. The fact that my title_case method uses capitalize is incidental, just as it's incidental that I prevent people from having too-short passwords by using validates_size_of. There are other ways to do these things, so it's really a matter of testing for functionality (I'll let someone else be the one to introduce the word 'behavior' into the thread :slight_smile: rather than implementation.

David

6

I find that kind of estimations hard to believe, because they sort of assume the alternative amount of work equals development minus tests.

But that is not the case, you need to sum all the bug fixing that comes afterwards, normally in production, and normally accompained by stress, and the uncertainty to deploy yet again without the safety net of a test suite, heh, perhaps you broke some other part of the system, and sometime you do.

Now, in frameworks or libraries you just can't afford publishing something without tests. People are relying on your work, they are using your software in _their_ production systems. So extra care is needed.

I got accostumed to the idea using (and humbly writing) Perl modules from CPAN. As you probably know, CPAN modules are expected to have a test suite. That test suite is run when the module is installed in _your_ machine, locally, which is even better because all I know as an author is they pass in my computer (and the ones volunteered by CPAN testers). If the test suite does not pass module managers won't install them unless forced.

-- fxn

7

> +1. Actually, *it* doubles the amount of work.

I find that kind of estimations hard to believe, because they sort of
assume the alternative amount of work equals development minus tests.

But that is not the case, you need to sum all the bug fixing that
comes afterwards, normally in production, and normally accompained by
stress, and the uncertainty to deploy yet again without the safety
net of a test suite, heh, perhaps you broke some other part of the
system, and sometime you do.

You're right. But if you want to test your code on boundaries conditions (which are the ones which most of time break your code), you'll need to imagine them first and then write your methods. It's just like the writing of your app's core. For your project, you have (let's assume you're not the project leader and you don't have to mess^W^understand what your customers really want) a pretty precise idea of what your code should do, and you spend time thinking how to implement it. For testing purposes, you already know how to implement tests since it's just like the code in your app's core, but you need to imagine all sort of conditions: normal cases, boundaries, weirdy, etc.

Actually, I didn't meant to say that's development process and testing require EXACTLY the same time, but testing process requires a lot of time.

Now, in frameworks or libraries you just can't afford publishing
something without tests. People are relying on your work, they are
using your software in _their_ production systems. So extra care is
needed.

I completly agree. I am really surprised whenever I read on blogs: 'hey! checkout my acts_as_hibernate plugin. INSTALL: svn co http://svn. & script/generate hibernate_plugin lots_of_messy_xml etc.'. What about rake test:plugins?

I got accostumed to the idea using (and humbly writing) Perl modules
from CPAN. As you probably know, CPAN modules are expected to have a
test suite. That test suite is run when the module is installed in
_your_ machine, locally, which is even better because all I know as
an author is they pass in my computer (and the ones volunteered by
CPAN testers). If the test suite does not pass module managers won't
install them unless forced.

'perl is dead' (© my algorithmic teacher)

8

> +1. Actually, *it* doubles the amount of work.

Based on the test writing that I've been doing the last few days, I'd say that writing tests turns a 'N' job into a '2N' job... possibly a '3N' job...

Perhaps I'm just new at it and will get faster with practice, but taking the time to think up tests takes at least as much time as it does to write the code. And then you have to debug the tests, because they are no more likely to work the first time than any other code. I've just spent a couple hours writing Unit tests for a handful of functions.

However...

But that is not the case, you need to sum all the bug fixing that comes afterwards, normally in production, and normally accompained by stress, and the uncertainty to deploy yet again without the safety net of a test suite, heh, perhaps you broke some other part of the system, and sometime you do.

I guess this is the point. An investment now will pay off in the future.

Additionally, in writing a bunch of 'pointless' model tests, I did find a bug in a migration... that is the model, and my tests, expected the database to be set up one way, but the test failed and thereby uncovered a bug that had been lurking in the database.

So testing the connection between the model code and the database is one use of validating things that you already "know" are correct.

-- John

9

Maybe just some viewpoint shift is needed? Take look at BDD and RSpec, write executable specification *before* code, so first you state:

context "Data Model " do    specify "should not allow blank names" do       @model.name = ''       @model.should_not be_valid    end end

and only then implement the spec:

class Model < ActiveRecord::Base    validates_presence_of :name end

10

Don't look it as testing Rails. You are testing your model on testing validates. Let's say you write this class:

class Person < ActiveRecord::Base   validates_length_of :phone_number, :within => 7..10   ... end

So you write a test:

def test_phone_number_length_correct   person = Person.new(:phone_number => '1234567')   person.valid?   assert !person.errors.invalid?(:phone_number), 'Valid Phone Number Coming Back as Invalid' end

I think we would all agree that this test makes sure that a valid phone number passes.

Now not thinking you do a find and replace and replace all 7s with 8s or maybe number with numbers is more likely. So you have:

class Person < ActiveRecord::Base   validates_length_of :phone_numbers, :within => 7..10   ... end

Now you can place :phone_number => '12' into your database because you are now validating :phone_numbers.

If you have a test you run tests after you completed the find and replace, and this error would be caught right away. That is why you 'test Rails.'

Amos King Ramped Media Inc./US Postal Service Programmer Analyst dirtyInformation.com

11

Amos.L.King wrote:

  person = Person.new(:phone_number => '1234567')   person.valid?

Now not thinking you do a find and replace and replace all 7s with 8s or maybe number with numbers is more likely.

Or you internationalize. :slight_smile:

(Good sample BTW!)

12

To me it seems like testing is duplicating (some times 3-4 times) you work; but in a good way.

So if I have the following:

class Person < ActiveRecord::Base   validates_length_of :phone_number, :within => 7..10   ... end

And the test:

def test_phone_number_length_correct   person = Person.new(:phone_number => '1234567')   person.valid?   assert !person.errors.invalid?(:phone_number), 'Valid Phone Number Coming Back as Invalid' end

I am basically saying in two separate places that my application requires phone numbers to be in a certain format. By putting this in two or more separate places (and using different syntax) I find myself significantly less likely to make a mistake (because I have to think about it in more than one way) and I find myself significantly more likely to find the errors.

If the odds of coding something wrong (i.e. differently than I think I am coding) are 20%. Then the odds of coding the same thing wrong twice are about 4% (0.2 * 0.2). Writing tests has made me go from writing code that does what I want 80% of the time to doing what I want 96% of the time.

I'm sure real life difference isn't as simple as that math, but I have found the concept to be true.

PS. I'm not saying that repetition within you application code is good. DRY is still the best way to go!!!!

13

Hi, it's simply about testing your application. I tend to have business login in the models. Thus, I don't really expect this to have been done by the core team because they do not know anything about login with my modules. I also I tend to perform both functional and integration testing as well.

Rails makes it very easy to perform testing at very levels of your application. If you think that it's pointless, then you should try testing J2EE and/or PHP application. When I wrote code in these technologies, I wished that there was a testing infrastructure like Rails.

Next, I tend to write my tests first. Then I write the code for my Rails application so that the tests pass. I'm still getting use to doing things this way as well as writing tests in general.

Finally, if your client knows enough about Rails, you may be require to provide these tests as part of the projtect.

-Conrad