rake test vs. rake test:functionals

I think I have a fundamental misunderstanding of what 'rake test:functionals' does for me.

I assumed it did the exact same thing for me that 'rake test' did, except that it only ran the functional tests. It seems like this was a bad assumption because in the project I'm working on, when I run 'rake test', I get no failures or errors, but when I run 'rake test:functionals', I get failures.

Is there something different about what these things do in terms of fixture loading or pre-test setup that I'm not seeing?

TIA...

Bryan

I don't have an answer for you, Bryan, but I too am curious about
rake-related testing differences. My problem is that if I run one of
my tests as follows

ruby test/unit/box_test.rb

it all passes, but if I run "rake test" or just simply "rake" the same
test fails.

Thanks in advance, whoever can shed light on this problem!
Ben

Here's what was wrong with mine.

The functional test that was failing when I ran 'rake test:functionals'
has a long list of fixtures it is using.

fixtures :f1, :f2, :f3

I looked closer at what the actual test method was doing, and realized
that it was dealing with a type of model object whose fixture did not
appear in the list of those I was including. So, I changed that line to
this:

fixtures :f1, :f2, :f3, :f4

... and all is well now.

So... that fixed my problem, but now I'm a little scared because
something else I assumed was happening a certain way seems not to be.

Since this particular functional test was passing when I simply ran
'rake test', I can deduce that one of my functional tests was including
the ':f4' fixture, and that data was still laying around in the
database, enabling that functional test to eventually pass.

This scares me because I assumed the fixtures would be loaded and wiped
away for each and every test so that the state of the database was
known. Is this a bad assumption?

Ben Nevile wrote:

Correcting something inline...

Bryan Noll wrote:

Here's what was wrong with mine.

The functional test that was failing when I ran 'rake
test:functionals' has a long list of fixtures it is using.

fixtures :f1, :f2, :f3

I looked closer at what the actual test method was doing, and realized
that it was dealing with a type of model object whose fixture did not
appear in the list of those I was including. So, I changed that line
to this:

fixtures :f1, :f2, :f3, :f4

... and all is well now.

So... that fixed my problem, but now I'm a little scared because
something else I assumed was happening a certain way seems not to be.

Since this particular functional test was passing when I simply ran
'rake test', I can deduce that one of my functional tests

scratch 'functional tests', should say 'unit tests'

Well for one thing, if you run 'rake test" all your unit tests run
before your functional tests. When you run 'rake test:functionals'
your unit tests don't run.

So I would suspect that your unit tests are loading something
(fixtures or whatever) that are not getting loaded when you run
functional tests alone.

That's the kind of thing I would look for first. Even try running your
tests individually and see if they pass. If they don't, you'll likely
find that your aren't loading something you need in the individual
test that is failing.

Thanks Robert. This is exactly what it turned out to be (see my
response in the thread about 3 minutes before yours). Do you have any
insight to my concern that I'm not getting a clean slate/known state
after unit tests have run?

Robert Walker wrote:

It appears to me from my testing that the test database tables are
emptied at the beginning of tests but are not emptied after the test
completes. Running functional tests that do not load the fixture in
question does not empty the databases table in question. Therefore, a
functional test reading data from a table where the fixture is not
loaded by that specific functional test will use the data that may
have been left behind from a previous test run.

Hope that's not too confusing. It basically boils down to ensuring
that you are loading fixtures for all tables accessed by that specific
test.

Ugg. And I thought that the builtin test framework was bad enough
already.

What this means is that your tests need to do complete teardowns--
including the test database. I've had a number of cases where I
mocked behavior for a test and then had the mocking interfere with the
testing of the real function. I tend to be pretty aggressive about
creating modules to test inside now. This sounds like another time
where best practice is going to have to be learned the hard way.

You've discovered that it's fixture-related. You have a test case that
is using a fixture that you aren't explicitly loading, but that a
different test case *is* loading. When you run the test cases in
different orders, you can get strange errors.

The solution is to specify *all* your fixtures in each test case. I
have the following method in test_helper.rb:

  def self.all_fixtures
    Dir.glob(File.join(File.dirname(__FILE__), 'fixtures', '*.{yml,csv}')).each
do |fixture_file|
      fixtures File.basename(fixture_file, '.*')
    end
  end

Then in each test case, instead of listing specific fixture names, I just use:

   all_fixtures

IMHO, this will land you right back in the soup. Different tests will
have different needs for fixtures. And edge testing conditions tend
(very much) to be incompatible with each other. Certainly, you might
have a base set of fixtures, or even a collection which might be
addressed as an array, but the topology is likely to be complex
overall.

YMMV and all, but you are far less likely to run into trouble if you
load precisely the fixtures that you intend for each test or set of
tests--and clean up when you are done.