Sounds sensible. Just wanted to check that was the general community's vibe.
Fred
Actually I’ve got a question first. What is the necessity of freezing time in the first place? If it’s done to
make all fixtures have consistent timestamps while loading, why can’t you just unfreeze it as soom asn fixtures are
objectized?
You could probably get away with it quite a lot of time. I've waffled a bit about this 钛刻专家——官网 The short version is that any time a test does something somehow connected to Time.now you can get failures (depending on how much time passed between the point at which you loaded fixtures and the point at which the test run), because previously it was true that (basically) Time.now = Time when fixtures were loaded = created of an object with created_at: <%= Time.now.to_s(:db) %>. I think (and certainly my experience when i wrote this with our apps) that maintaining that assumption is helpful (and certainly outweighs any disadvantages I found).
Fred
I've been using the same technique for speeding up my tests, independently developed. I don't think freezing time is something that should be enforced by rails, if some of your tests really need it then those tests should do it.
For me freezing time has not really been a problem because the timestamps that I do test are usually specified with a fixed hour/minute part but a changing date (fixed offset against today) date part.
So I think this patch in general is great, but the time freezing thing should not be included.
[questions about the point of freezing time snipped]
You could probably get away with it quite a lot of time. I've waffled a bit about this 钛刻专家——官网 with-parsimony/
The fact that this assertion:
assert_not_equal(Time.now.to_i, (sleep 1; Time.now.to_i))
will always fail is a little bit disconcerting. Our codebase has a fair amount of time-based tests. While most of them are fairly chunky, e.g.,
Time.travel_to(2.weeks.from_now) do assert_something_date_sensitive end
...some of them do depend on time continuing to run. In fact, we've carefully made sure that time continues to run even when we've temporarily changed the definition of 'now'.
It seems like the window of danger if time continues to run during a test is relatively small. Additionally, I'm not convinced that a test which depends on the timestamps of fixture data to a granularity of less than, say, five minutes is suitably robust.
~ j.
...some of them do depend on time continuing to run. In fact, we've carefully made sure that time continues to run even when we've temporarily changed the definition of 'now'.
I'm curious - what sort of things depend on time continuing to run ? Would your tests still pass if say you replaced your test machine with one that ran 10 times faster (or say if you switched to a faster implementation of ruby)
It seems like the window of danger if time continues to run during a test is relatively small. Additionally, I'm not convinced that a test which depends on the timestamps of fixture data to a granularity of less than, say, five minutes is suitably robust.
Rather than window of danger what I was really gunning for is consistency. We've also got our share of time-based tests, but they do tend to be on the shorter timescale of things.
It certainly wouldn't be hard to only freeze time for the purpose of loading the tests, providing the option to freeze it permanently if desired. The other question is what will break more tests out there: time that doesn't pass or fixtures data that is older than existing tests expect ? The apps I've worked on couldn't care less about the former, but I've no idea what the general pixture is
Fred
Well I’d say my tests take the assumption that time is never frozen, and make their ways
accordingly (create an object, see if it’s timestamps make sense, see if the time calculation makes sense).
The fact that time is changing when the system is running adds to the realism of the test environment IMO.
Will I for one always assume that even if I put <%= Time.now %> in the fixtures this will get evaluated somewhat before
my real test will get to work. I never rely on Time.now being frozen, that is (and consider it a bad practice, except for some very specific,
surgical cases). I don’t think freezing Time in tests is something to do to guarantee consistency - if you do time-based testing you just gotta do it
right. If you have it about borderline cases, such as “if something expires after date X, and we load a page after date X, what would then happen?”
I make a fixture that already expired by putting <%= 20.days.ago.to_s(:db) %> in the fixture. And test it accordingly.
+1, I also experimented with freezing time and was not impressed.
Sure that's the best practice. However (as far as I can tell) if you have been following best practices then freezing doesn't add problems, but if you haven't then the whole only loading fixtures thing once might break some tests. So do you go a little out of your way to accommodate these people, do you ignore them and let them fix their tests when they upgrade or do you toss a little vinegar in their eyes by requiring them to set some setting ?
Fred
Sure that's the best practice. However (as far as I can tell) if you
have been following best practices then freezing doesn't add
problems, but if you haven't then the whole only loading fixtures
thing once might break some tests.
I think we should leave freezing time out as a separate concern and just focus on the fixture speedups for now. No need to marry the two issues, imo.
I think you still want to do enough freezing so that it looks like your fixtures were loaded at the same time. Otherwise you end up with weird stuff like blog posts created before the user that created them & other oddities.
Fred
I don’t think making sure that fixtures loading at the same time is the role of the faster_fixtures code. If there is a fixture created at Time.now that depends on another fixture created at Time.now then clearly the developer doesn’t care about comparing the creation dates.
Better ways to avoid those ‘oddities’ would be to load fixtures in a reasonable order or to set more plausible created_at datestamps. Or we could have a separate discussion about freezing time.
Aside from the mocha dependency and the time freezing I think this is golden.
::Jack Danger
On 9/25/07, Frederick Cheung <frederick.cheung@gmail.com> wrote: On 26 Sep 2007, at 00:35, DHH wrote: > >> Sure that's the best practice. However (as far as I can tell) if you >> have been following best practices then freezing doesn't add >> problems, but if you haven't then the whole only loading fixtures >> thing once might break some tests. > > I think we should leave freezing time out as a separate concern and > just focus on the fixture speedups for now. No need to marry the two > issues, imo.
I think you still want to do enough freezing so that it looks like your fixtures were loaded at the same time. Otherwise you end up with weird stuff like blog posts created before the user that created them & other oddities.
I don't think making sure that fixtures loading at the same time is the role of the faster_fixtures code. If there is a fixture created at Time.now that depends on another fixture created at Time.now then clearly the developer doesn't care about comparing the creation dates.
Just talking about my code for a second, that's not quite true: I don't care about the specific values of any of the created_at's in the database, I only care about the relations between them: what happens before/after/at the same time at other times. Hard coding '2007-04-08 19:00:00' and '2007-04-08 19:02:00' in fixed dates seemed a lot nastier then having one thing created_at 2.minutes.ago and the other at Time.now.
Better ways to avoid those 'oddities' would be to load fixtures in a reasonable order or to set more plausible created_at datestamps. Or we could have a separate discussion about freezing time.
Is there such a thing as reasonable order? I'm not convinced things couldn't be a bit circular. But at the end of the day I think it holds that the 2 are orthogonal
Fred
Aside from the mocha dependency and the time freezing I think this is golden.
I scrapped the mochaness anyway
That’s what I was talking about. When eval’ing the fixture code, freeze, then unfreeze.
Couldn't that happen in production as well, though? Obviously not this particular case, because creating a user generally happens a few steps before a post is made. But for any two related records, unless your test assumes which order they're written in, one could be written during the last jiffy of 11:22:33, and the other could be written during the first jiffy of 11:22:34.
In fact, let's say this is a blog that allows anonymous posts, but for recordkeeping, it automatically creates a user at post time if one doesn't already exist. That could easily lead to the above situation.
The problem with freezing time is that it assumes that nobody else is doing anything with it. I often test expirations by mocking Time.now; will your time-freeze interfere with my mocking, no matter whether I'm mocking in FlexMock or Mocha or RSpec?
Otherwise you end up with weird stuff like blog posts created before the user that created them & other oddities.
In fact, let's say this is a blog that allows anonymous posts, but for recordkeeping, it automatically creates a user at post time if one
doesn't already exist. That could easily lead to the above situation.
Sure, but that;s not quite the point of my example: the point was
that if you have a situation where logically x.created_at must be >=
y.created_at then that no longer necessarily holds (i'll give an
example from my app: we have questions and answers (that arrive and
are sent back over sms) It's just plain nonsensical for the answer to
be sent before the question it's answering arrived). To an extent
this could already happen in rails: fixtures are all loaded in one
go, but obviously they're not all guarenteed to be loaded within the
same secon.
Fred
The problem with freezing time is that it assumes that nobody else
is doing anything with it. I often test expirations by mocking Time.now;
will your time-freeze interfere with my mocking, no matter whether I'm
mocking in FlexMock or Mocha or RSpec?
I'm not sure about that one. I've often changed time more than once
in a test, once for the fixtures stuff and then refrooze it again in
the tests themselves. That was with mocha. With this code' i'd
changed this so that it reopens Time and alias_method_chain's now. I
think that makes it orthogonal to mocha et al but I don't actually know.
Anyway, the time changing stuff looks to be a separate problem for now.
Fred
I’m sorry I’m coming to this conversation rather late in the day, but I wonder whether there has been any thought given to moving fixtures from core Rails into a plugin? I know that a lot of developers (including me) don’t use them in their tests any more - preferring instead to use mocking in unit tests that don’t touch the database in combination with integration tests that do touch the database and explicitly call ActiveRecord create!, etc, to insert data. I only mention the idea because seems to fit with the philosophy of moving stuff out into plugins. Anyway, just a thought.
Well apart from anything else the activerecord tests themselves use fixtures quite a lot and obviously mocking out the database interactions for those tests is probably not appropriate since that's what is being tested. We still use fixtures a lot (but then on average our database interactions are quite a bit beyond your typical CRUD, so eg we want test coverage for our hand rolled SQL etc...).
Fred
Even though I’m still using fixtures in most places I think the idea has merit. I’d be against removing them though just because it would be one more hurdle keeping new developers from testing. It’d be nice to keep a nice, low ramp into teaching the basic testing practices.
::Jack Danger