fixtures, factories and the standard testing framework.

rubyonrails-talk
1

A little background: I've been enjoying rails for about a year now tinkering on meaningless projects here solely for the purpose of learning the framework... not because I actually needed to use those apps for anything serious. Now that I started writing my first serious app, naturally I want to dive into test driven development ("finally" I know - please, no harassment - I'm trying). Now the way I see it, the standard testing framework is more than sufficient for my needs and as time goes on it's only going to improve. So while things like RSpec and Cucumber are all well and good - they don't really fit my needs.

With that out of the way - I dove right into creating fixtures and -- halt. Fixtures, while improving - are still cumbersome for nested models. Take for example my Quote model - it has_many QuotableOperations and QuotableQuantities. When I test them I want to have complete associations... Fixtures don't seem to like assigning to has_many unless I do it from the belongs_to side (which feels unnatural) so I gave up on fixtures and started working with Factories (Factory Girl)... Now, a few days into Factory Girl and it also seems too cumbersome for the job at hand. I'm not sure, but it seems like either:

*I'm doing it wrong*

or

*There's a better way*

Can anyone point me towards the light?

Take this very simple test for example:

I have a

Quote(:description => "Some job to quote") - with one QuotableOperation(:operation_name => "Engineering") - and two QuotableQuantities(:quantity => "5", :rate => "0.21") & QuotableQuantities(:quantity => "10", :rate => "0.21")

:description, :quote_operations and :quote_quantities are validate_presence_of'd on the Quote model.

SO - a simple test like:

test "the description of the quote is not blank" do     ***test*** end

seems to be more trouble than it's worth because I need a fully associated Quote object.

A helping hand in the right direction would be more than appreciated.

Thanks, - FJM

2

C'mon guys, I know this problem can't be that hard to tackle. Is there anyone out there with some helpful insight?

3

frankjmattia@gmail.com wrote: [...]

Now the way I see it, the standard testing framework is more than sufficient for my needs and as time goes on it's only going to improve.

Not necessarily. I don't use Test::Unit, so I could be wrong (and someone please correct me if I am), but it seems that all the important work these days is being done with RSpec and Cucumber, and Test::Unit has more or less stagnated (with a couple of exceptions, such as Phlip's plugins).

So while things like RSpec and Cucumber are all well and good - they don't really fit my needs.

Try them out before you say that.

* The reason I don't use Test::Unit is that RSpec has a syntax and philosophy that I like much better. Test::Unit feels clunky and hard to use by comparison.

* Cucumber, however, is a completely different animal, complementary to RSpec/Test::Unit. It takes a little while to figure out how to use it, but once you do, you'll wonder how you lived without it. Please don't ignore it.

With that out of the way - I dove right into creating fixtures and -- halt. Fixtures, while improving - are still cumbersome for nested models. Take for example my Quote model - it has_many QuotableOperations and QuotableQuantities. When I test them I want to have complete associations... Fixtures don't seem to like assigning to has_many unless I do it from the belongs_to side (which feels unnatural)

I don't find this to be a problem, but I agree that fixtures are clumsy and hard to work with once you get a serious test suite going. Use factories.

so I gave up on fixtures and started working with Factories (Factory Girl)... Now, a few days into Factory Girl and it also seems too cumbersome for the job at hand. I'm not sure, but it seems like either:

*I'm doing it wrong*

or

*There's a better way*

Can anyone point me towards the light?

Try Machinist.

Take this very simple test for example:

I have a

Quote(:description => "Some job to quote") - with one QuotableOperation(:operation_name => "Engineering") - and two QuotableQuantities(:quantity => "5", :rate => "0.21") & QuotableQuantities(:quantity => "10", :rate => "0.21")

:description, :quote_operations and :quote_quantities are validate_presence_of'd on the Quote model.

SO - a simple test like:

test "the description of the quote is not blank" do     ***test*** end

seems to be more trouble than it's worth because I need a fully associated Quote object.

With Machinist, once you set up the blueprints (which you only have to do once), Quote.make will give you a fully associated Quote.

A helping hand in the right direction would be more than appreciated.

Good luck! Please ask if you have further questions.

Thanks, - FJM

Best,

4

C'mon guys, I know this problem can't be that hard to tackle. Is there anyone out there with some helpful insight?

Don't come down too hard on everyone for not answering right away.
Type "rails test factory" into Google and you can do some independent
research.

I personally use Machinist and Faker (good article: http://toolmantim.com/articles/fixtureless_datas_with_machinist_and_sham) . Others swear by Factory Girl (read: Companies like yours create meaningful solutions with thoughtbot)   and still love Object Mother (read: Object Mother Testing Pattern in Rails — Jeff Perrin) . You may even like Fixjour (read: Elctech.com is for sale | HugeDomains) .

A little background: I've been enjoying rails for about a year now tinkering on
meaningless projects here solely for the purpose of learning the framework... not because I actually needed to use those apps for anything serious. Now that I started writing my first serious app, naturally I want to dive into test driven development ("finally" I know - please, no
harassment - I'm trying). Now the way I see it, the standard testing framework
is more than sufficient for my needs and as time goes on it's only going to improve. So while things like RSpec and Cucumber are all well and good - they don't really fit my needs.

There are far more lively communities surrounding rSpec and Cuke, so
if you want help, you'll find a lot of people willing to answer
questions. I personally feel the thinking behind spec first makes more
sense than test first. At the execution level it's the same, but what
you are writing is a spec for code that doesn't yet exist, not a test.
Cucumber is a whole level of goodness that is orthogonal (in most
ways) to either rSpec or Test::Unit. Acceptance testing can be a
lifesaver. You'd be surprised how often a good set of stories can
either flush out poorly designed features or save your bacon if you
break something.

With that out of the way - I dove right into creating fixtures and -- halt. Fixtures, while improving - are still cumbersome for nested models. Take for example my Quote model - it has_many QuotableOperations and QuotableQuantities. When I test them I want to have complete associations... Fixtures don't seem to like assigning
to has_many unless I do it from the belongs_to side (which feels unnatural) so I gave up on fixtures and started working with
Factories (Factory Girl)... Now, a few days into Factory Girl and it also seems too cumbersome for the job at hand. I'm not sure, but it seems like either:

*I'm doing it wrong*

or

*There's a better way*

Most will concede that fixtures that describe associations are brittle
and coupled too tightly to a particular implementation. That's why the
fixture replacements are so popular. Also consider using a mocking/ stubbing framework. rSpec comes with one baked in, or you can use
Mocha or FlexMock. Best to only mock and stub in unit tests, as
acceptance tests are supposed to exercise the whole stack. Of course,
if your app needs to reach out over the network or something like
that, you might have to use something like FakeWeb or a mock/stub for
the service you don't want to hit with your test.

5

I most certainly appreciate any and all replies and with that in mind, I hope I didn't come off as coming down hard... Intonation is a hard thing to communicate in textual media... Anyhow, sorry for sounding that way. Perhaps an emoticon could have livened it up a bit... Hehe.

Anyway...

I've just ripped out all of my Test::Unit stuff am having a go at starting fresh with RSpec, Cucumber and Machinist... Please understand that I've only been casually programming for the past 10 years as a way to kill time and challenge myself. I have no formal training, never wrote anything serious and the fact that I've written anything at all that works is a blessing because before last week - I had never written a test in my life - let alone put any thought into testing an entire application. TDD/BDD are both intriguing and believe me - I see their advantages... I'm just very slow on the uptake right now. It's not the way I'm used to programming **read that as "I've developed a lot of bad habits on my own and now I want to break them the right way**.

SO.

The one thing that keeps coming back to haunt me is that no Factory framework seems to let you test complete objects from the has_many side of the association without doing some serious legwork. I see examples on the Machinist git page that suggest using helper methods but that brings me to another road block. Where do I write them? How are they called? What types of things should I not be testing at that level?

Now, my own personal itch of a project is still small enough that I don't mind ripping entire sections of tests out and rewriting them with new frameworks just to learn how. 1) I have the time and 2) I'm interested in learning how to do it right more than I am in doing it... If that makes sense. The parts of my app work just fine for me without tests... but knowing my itch, I'm going to want to add complexity and knowing how Ruby and Rails works - I want to add it the right way so that it's fun for me to do...

Anyway - I'm getting off topic again.

Back in my first post I describe a situation that I was trying to test. Could you tell me how I should be approaching that issue (which may very well be much different than the way I've been approaching it). Why is testing from the has_many side so taboo? Is there a legitimate way of doing it from the belongs_to side?

Thank you for your reply and thank you too Marnen.

You have both given me good food for thought, even if I don't know how to eat it yet.

Hehe. Thanks, - FJM

6

In retrospect that was a bad choice of words. What I should of said was, "make complete objects". Then again, I may not fully understand what I'm trying to do... In which case, this may be a long, painful road... err. I mean... labor of love.

From the machinist webpage.

7

Hello--

I think you are asking a lot of good questions. I would direct you to
the rspec and cukes Google Groups for a great in-depth discussion of
all this stuff. Also, Pragmatic Programmers has a beta book called
"The rSpec Book" (IIRC) and you can buy it and read the PDF. The final
print copy (if you buy one) will be delivered when it is complete.
This book covers both rSpec and Cucumber and shows a good division of
responsibility between the two tools.

As I am really sold on rSpec and Cucumber, this is kind of OT for the
Rails list, but my recommendations are:

- Don't try to re-test Rails. If there was a bug in
validates_presence_of or has_many, it would probably be known. - Corollary: Test *your use* of Rails. So (e.g.) if you use
validates_format_of, make sure your regex is right. - Work from the outside in. Describe a feature (Cucumber), make it
fail, drive the feature out, diving into rSpec where unit tests will
be of value. - [controversial] Don't write controller tests if you can avoid it.
Write Cucumber features. - [less-controversial] Make your controllers so stupid-simple that you
can unit-test your models and get good coverage. - Re-evaluate your progress as you do the red-green-refactor cycle for
a while. What works for some won't work for all. - Keep your eyes open for what others are sharing about the tools you
are using and new tools that also might help. - Absolutely, keep the big picture in sight: You aren't writing specs,
features, or tests; you're writing an app.

Good luck... a few more answers below:

Frank J. Mattia wrote:

[...]

I've just ripped out all of my Test::Unit stuff

That may or may not be good, depending on how much you had to rip
out. Then again, you can always use your version control system to go
back...

I am having a go at starting fresh with RSpec, Cucumber and Machinist... Please
understand that I've only been casually programming for the past 10 years as a way to kill time and challenge myself. I have no formal training,

Neither do I.

never wrote anything serious and the fact that I've written anything at all that works is a blessing because before last week - I had
never written a test in my life - let alone put any thought into testing
an entire application.

Hey, don't feel bad. I worked professionally as a programmer for
years without a systematic testing approach.

TDD/BDD are both intriguing and believe me - I see their advantages... I'm just very slow on the uptake right now. It's not the way I'm used to programming **read that as "I've developed a lot of bad habits on my own and now I want to break them the right way**.

OK.

SO.

The one thing that keeps coming back to haunt me is that no Factory framework seems to let you test complete objects from the has_many side of the association without doing some serious legwork.

What are you talking about? What do you mean by testing from the has_many side?

In retrospect that was a bad choice of words. What I should of said was, "make complete objects". Then again, I may not fully understand what I'm trying to do... In which case, this may be a long, painful road... err. I mean... labor of love.

From the machinist webpage.

================ Other Associations For has_many and has_and_belongs_to_many associations, ActiveRecord insists that the object be saved before any associated objects can be saved. That means ***you can't generate the associated objects from within the blueprint.***

The simplest solution is to write a test helper:

def make_post_with_comments(attributes = {}) post = Post.make(attributes) 3.times { post.comments.make } post end Note here that you can call make on a has_many association. (This isn't yet supported for DataMapper.)

So suppose I write the following:

def make_complete_quote_object(attributes = {}) quote = Quote.make(attributes) 5.times {    quote.quotable_operations.make } quote.quotable_quantities.make

quote end

Where does that go

I often put these in spec/spec_helper.rb. You may want to refactor
them out at some point so you can reuse them in your Cucumber steps.

and do I just use that in lieu of Quote.make? For example:

new_quote = make_complete_quote_objects(:description => "blah blah blah")

Yes. But... the make method is not the only one you will use. You may
find yourself creating partial objects look at make_unsaved and plan.
These will allow you to satisfy certain validation requirements that
simply can't be gotten from a factory.

8

I think between you and Marnen I've really discovered a lot of useful things.

Now that I'm starting to find myself sold on Cuke and RSpec I'm looking back on earlier today when I actually bought that book you mention. I'm a huge fan of the pragprog guys and always find myself looking towards their new books.

Whats more is, just 15 minutes ago I got my first feature to pass. I was pulling that code from the machinist git page and after a while, discovered the error with make and make_unsaved... The tutorial on their wiki has a mistake it seems.

Feature: Manage Quotes   I want to create and manage quotes

  Scenario: Quotes List     Given I have quotes with titles Threadmill, Trepan     When I go to the list of quotes     Then I should see "Threadmill"     And I should see "Trepan"

It only took a wee bit of fighting and determination but it's most certainly a start.

Again, your guidance has been very much appreciated.

I'm still a little weak in the knowing-what-to-test when area but "not testing the framework" seems so obvious now it almost hurts.

So now I have to learn what to test where... for example - looking at the big picture (feature) then diving into rspec for the unit tests. I still don't see the clear line. Also, from what I've read - to say its controversial to leave out controller tests is an understatement of understatements ... heh... It almost feels sinful to read it.

I'm going to start checking the cuke and rspec lists from now on - I can definitely see how this has gotten way off topic for rails but I most certainly thank you for the insight provided in light of that. Looking back I'm surprised I didn't get any "go to the right group cause we can't help you" responses.

Thanks again, - FJM

9

Hi   And none of you here specified

     a framework Which extends Test::Unit . So I think not a very big learning curve needed for a Ruby programmer unlike some other tools you specified

Sijo

10

Thanks for the link. I hadn't checked out shoulda as deeply as I could have. However, in all honesty - I really do believe I was looking for a paradigm shift from away standard testing. I haven't ruled out investigating it in the future, but for now I'm just diggin' rspec/ cuke/machinist.

And yes, while the learning curve is quite steep - the rails community has built up a huge amount of "accessible" knowlege (wikis, screencasts, github). It's almost feeling like a smooth ride.

11

If you're a shoulda fan, check out carlosbrando-remarkable (http://github.com/carlosbrando/remarkable/tree/master). It is an implementation of (AFAICT) the shoulda macros for rSpec. I used remarkable for controller specs, but I'm a far bigger fan of Cuke/Webrat in place of controller specs (heresy again!). Again, this is just me, but I put so little logic in the controller that a Cucumber step can normally cover it.

12

Steve Ross wrote: [...]

Also consider using a mocking/ stubbing framework. rSpec comes with one baked in, or you can use Mocha or FlexMock. Best to only mock and stub in unit tests, as acceptance tests are supposed to exercise the whole stack.

I find that since I started using Machinist, I've completely stopped using mock_model.

Of course, if your app needs to reach out over the network or something like that, you might have to use something like FakeWeb or a mock/stub for the service you don't want to hit with your test.

Yeah. Phlip (whom I generally consider a testing guru) suggests that's that's the only thing to mock; then again, he also uses fixtures and dislikes RSpec. :). OTOH, Martin Fowler (I think), whom I also tend to trust, says to mock everything that isn't what you're directly testing. Probably the best approach lies somewhere in between, but I don't know where...

Best,