timestamped migrations

rubyonrails-core
1

Hello all,

Pratik suggested I send an email to this list to discuss making timestamped migrations optional. I have submitted a ticket which includes a patch. I am a bit confused why the ticket was marked as incomplete as the solution seems rather simple. IMHO, making this optional would have saved the effort of removing code.

So, here’s my take:

From a single developer/small team perspective timestamped migrations were unnecessarily added as the standard format. The problem addressed by introducing timestamped migrations is a communication issue that doesn’t apply to everyone using Rails. I disagree with making timestamped migrations mandatory instead of optional. I also think the default should be the original numbering scheme due to legacy applications, not timestamps as Pratik says in the ticket. If people really wanted timestamped migrations, they could have added the plugin.

It’s the small things that annoy, here are a few examples:

  1. I use Terminal a lot and sometimes copy a migration to start a new one. The timestamp numbering makes this a pain.
  2. I have a project, Lockdown, that generates 5 migrations. I had to modify the next_migration_string method to add a sleep(1) call to prevent all migrations from having the same number. Big deal right? Well, timestamped migrations forced me to update Lockdown just because of this change.
  3. My “old” 2.0.2 app that had a bunch of migrations were now mixed with timestamped migrations. My migrations are no longer ordered in my file view.

There are other minor issues, but don’t think there’s a need to get that picky, this should be example enough.

In the future, should I submit a ticket first or just post something to this list?

Thanks for all the work, Rails saved me from the Java world. :slight_smile:

I hope timestamped migrations are made optional in the next release of Rails, but if not, I have a way around it. I’m writing this for other Rails users more than myself. -andy

2

Pratik suggested I send an email to this list to discuss making timestamped migrations optional. I have submitted a ticket which includes a patch. I am a bit confused why the ticket was marked as incomplete as the solution seems rather simple. IMHO, making this optional would have saved the effort of removing code.

It's marked incomplete because you've got the defaults backwards. UTC migrations should be the default:

+ # Specify whether or not to use timestamps for migration numbers + cattr_accessor :timestamped_migrations , :instance_writer => false + @@timestamped_migrations = false

There are other minor issues, but don't think there's a need to get that picky, this should be example enough.

I like the idea of the configuration option, just flip the default and get some feedback from other potential users to make sure it makes sense to them.

In the future, should I submit a ticket first or just post something to this list?

Generally a ticket is fine if it's a bug fix or something 'obvious'. But if you want to add an option or get some ideas about a new API or have a change in mind that might be controvercial, then this list should be your first port of call.

3

It’s marked incomplete because you’ve got the defaults backwards. UTC

migrations should be the default:

  • Specify whether or not to use timestamps for migration numbers

  • cattr_accessor :timestamped_migrations , :instance_writer => false

  • @@timestamped_migrations = false

If you don’t mind, could you explain why the should be the default? Or is this explanation already out there on a blog or something, I’m obviously missing something because I don’t get that reasoning.

thanks, andy

4

If you don't mind, could you explain why the should be the default? Or is this explanation already out there on a blog or something, I'm obviously missing something because I don't get that reasoning.

For all the reasons it was included in the first place, fewer conflicts on branches, etc. But most importantly, we don't want to change this behaviour back in a minor point release. Those who want to switch to integers again can do so easily enough, and we keep the timestamps for people who need them.

5

I’m not familiar with what the “etc” represents and find it funny you mention “Those who want to switch to integers again can do so easily enough”. When those who wanted timestamps could have done it easily enough without it being forced upon those who don’t.

You guys made the logical push of removing features from core that were “edge use cases” to plugins but then pulled a plugin into core and made the standard, without an option. That is quite confusing.

I understand the notion of “convention over configuration” and “opinionated software” but this is a step beyond.

6

Every application that gets coded by more then one person has benefit
with timestamped migrations. I think most Rails applications get
developed by > 1 person, so timestamped migrations are the logical
choice imo.

regards, Jan De Poorter http://workswithruby.com

7

Every application that gets coded by more then one person has benefit

with timestamped migrations. I think most Rails applications get

developed by > 1 person, so timestamped migrations are the logical

choice imo.

Thanks for the input Jan. I’m not saying timestamps don’t have benefits, I just disagree with the way they were introduced and the fact they were made mandatory. Git has benefits for a team environment, but not everyone is forced to use it, it’s an option.

8

Whether you're on git, svn, cvs, copying files to one-another, numeric
migrations are a show-stopper.

To revert the question, why do you want numeric migrations that bad?

9

I'm not familiar with what the "etc" represents and find it funny you mention "Those who want to switch to integers again can do so easily enough". When those who wanted timestamps could have done it easily enough without it being forced upon those who don't.

The timestamped migrations are fully backwards compatible with the integer based ones, there are no practical downsides to switching to it beyond the Timestamp collision that you've mentioned. There are however some aesthetic considerations which I think would be nice to handle with an option like you've mentioned. I personally think they look hideous and am grateful that you've taken the time to whip up the option.

But we're not going to flip the default back just to satisfy those aesthetic concerns and won't take a patch which does that.

10

The timestamped migrations are fully backwards compatible with the

integer based ones, there are no practical downsides to switching to

it beyond the Timestamp collision that you’ve mentioned. There are

however some aesthetic considerations which I think would be nice to

handle with an option like you’ve mentioned. I personally think they

look hideous and am grateful that you’ve taken the time to whip up the

option.

But we’re not going to flip the default back just to satisfy those

aesthetic concerns and won’t take a patch which does that.

Since you replied while I was replying to Jan, I’ll just put both here:

to Jan:

It’s not about wanting numeric migrations “that bad” (I can get around it), it’s more about the way the change was introduced and the lack of an option. You would have ended up with fewer code changes by making it an option so it’s not like it was a great deal of work. My fix to this was simply a revert of the 2.1 commit with a few conditional lines of code added.

I still haven’t gotten a good reason to make it mandatory other than the “working in a team environment benefit” and the sense it’s a “because we said so” attitude. To me, this is similar to the semicolon in the path idea. By the way, that was reverted on a bug fix release: 1.2.4 so I don’t get the argument about changing this on a minor release.

to Michael:

I expected it wouldn’t be accepted with the default I proposed, I was just looking for some explanation for the initial change.

Keep in mind, it was Pratik’s idea to start this thread, not mine. :slight_smile: But I did enjoy the back-and-forth.

I’ll whip up another patch when I have the time.

thanks, andy

11

I still haven't gotten a good reason to make it mandatory other than the "working in a team environment benefit" and the sense it's a "because we said so" attitude.

With your patch it won't be mandatory any more, which is the best of both worlds

To me, this is similar to the semicolon in the path idea. By the way, that was reverted on a bug fix release: 1.2.4 so I don't get the argument about changing this on a minor release.

It's not really anything like the semi colon issue as it functions just fine. We have all the benefits of the old style of migrations, with some nice branch-friendly properties.

Beyond the ugly file names are there any downsides which I'm missing?

to Michael:

I expected it wouldn't be accepted with the default I proposed, I was just looking for some explanation for the initial change.

Keep in mind, it was Pratik's idea to start this thread, not mine. :slight_smile: But I did enjoy the back-and-forth.

I'll whip up another patch when I have the time.

Look forward to it.

12

My understanding is that the problem that timestamped migrations is the solution to is that of multiple developers committing the same numbered migration file. This, as far as I can tell, is a problem best solved not by the framework, but with intra-team communication: “Hey everybody, I’m creating and checking in migration XXX now, update your code before you create another.” it’s slightly clumsy sure, but the benefits of increased team communication outweigh that awkwardness: “why are you committing a new migration?” “to add ABC to model XYZ” “oh, wow that’s awesome” “yeah, I know”.

The timestamp change “solves” the problem by making it much more unlikely that collisions will happen. Because the collision is less likely to happen I can see a (admittedly super pessimistic and extreme) scenario. There’s a team with very little communication and when there is a collision they freak out because they don’t know what to do what with collisions being an uncommon event).

As to a numeric or timestamped (or something entirely different) migration naming scheme being the default I don’t really mind (both have advantages in different scenarios), but an option to choose the mechanism get my vote every time.

Muz

13

(Josh Susser, I copied you because this could be a good opportunity to slip migration concordance logic into core. I'd be glad to help.)

The funny thing about all of this, is that UTC migrations were introduced as a quick way to fix the problem of migration number clashes. It was easier than some of the other solutions, which would've involved changing the way schema_info works.

But then we changed schema_info anyway, to allow for interleaved migrations (UTC or not). Now that schema_migrations exists, one can thing of other possibilities.

For example, we could track, instead of the timestamp or number, the complete migration name (or a hash of it). That way, we could have multiple migrations with the same UTC or number, still have order, still have interleaved migration support, and never[*] have collisions.

The tricky part is to do this in a backwards compatible way. It's also important to note that many of the ideas listed here suffer from bitrot in the schema_migrations table, which might need garbage collection from time to time (as hashes change).

[*] Identical filenames would clash. But the only solution to that is to hash the entire migration file[&], which some people have suggested in the past (Josh, are you still reading?), to detect when migrations change, and what have you. Not that there's anything that can be done automatically in that case, but, at least the developer finds out of the discordant migrations, and can act on it.

[&] This can get tricky too. Differences in line endings (CR, LF, CRLF) could make life hell. And if you chose to ignore those, the day a change in line endings fixes a migration, you have an even trickier case of hell.

14

Beyond the ugly file names are there any downsides which I’m missing?

My understanding is that the problem that timestamped migrations is the solution to is that of multiple developers committing the same numbered migration file. This, as far as I can tell, is a problem best solved not by the framework, but with intra-team communication: “Hey everybody, I’m creating and checking in migration XXX now, update your code before you create another.” it’s slightly clumsy sure, but the benefits of increased team communication outweigh that awkwardness: “why are you committing a new migration?” “to add ABC to model XYZ” “oh, wow that’s awesome” “yeah, I know”.

This is a good example and one option, the other is maintaining a version file in the db/migrate directory that could be updated with the most current version. This “version” file could be committed (to notify team members) even if the migration is not ready to be committed. This is just a simple way to use a development practice to address this issue.

The timestamp change “solves” the problem by making it much more unlikely that collisions will happen. Because the collision is less likely to happen I can see a (admittedly super pessimistic and extreme) scenario. There’s a team with very little communication and when there is a collision they freak out because they don’t know what to do what with collisions being an uncommon event).

As to a numeric or timestamped (or something entirely different) migration naming scheme being the default I don’t really mind (both have advantages in different scenarios), but an option to choose the mechanism get my vote every time.

Muz

Yes, options are nice.

15

(Josh Susser, I copied you because this could be a good opportunity to slip migration concordance logic into core. I’d be glad to help.)

The funny thing about all of this, is that UTC migrations were introduced as a quick way to fix the problem of migration number clashes. It was easier than some of the other solutions, which would’ve involved changing the way schema_info works.

But then we changed schema_info anyway, to allow for interleaved migrations (UTC or not). Now that schema_migrations exists, one can thing of other possibilities.

I thought the ability to interleave migrations was a great addition and is partially responsible for my confusion for the timestamped requirement. I can now say, “I’m using migration 007, please use 008” and when 007 is committed the next db:migrate will pick it up. Nice.

For example, we could track, instead of the timestamp or number, the complete migration name (or a hash of it). That way, we could have multiple migrations with the same UTC or number, still have order, still have interleaved migration support, and never[*] have collisions.

The tricky part is to do this in a backwards compatible way. It’s also important to note that many of the ideas listed here suffer from bitrot in the schema_migrations table, which might need garbage collection from time to time (as hashes change).

[*] Identical filenames would clash. But the only solution to that is to hash the entire migration file[&], which some people have suggested in the past (Josh, are you still reading?), to detect when migrations change, and what have you. Not that there’s anything that can be done automatically in that case, but, at least the developer finds out of the discordant migrations, and can act on it.

[&] This can get tricky too. Differences in line endings (CR, LF, CRLF) could make life hell. And if you chose to ignore those, the day a change in line endings fixes a migration, you have an even trickier case of hell.

I really haven’t given much thought to a completely different approach because I thought the simple sequence worked well. However, maybe there is a better solution.?

16

Is there a way to address some of the less-than-perfect aesthetics of the UTC migrations without losing any of their (very useful) functionality?

Along the lines of what Jordi was saying, or perhaps setting the UTC stamp inside the migration itself rather than as part of the filename (I haven't looked at the code; would this require loading every migration every time you want to migrate?).

The only other problem (IMHO) with UTC migrations is the "reference problem".

I can't say to another developer on my team - "Oh yeah, migration 95 is failing on production", I have to say "that add_whatever_to_something" migration is failing.

Its the same with Git commit hashes actually :frowning:

17

The fact that there are two caveats to the theoretical approach are probably a testament as to how useful it will actually be.

I think the UTC timestamps are great. Someone earlier said this should be solved by communication, but then people are now saying that it's a problem that people have to communicate TOO MUCH by telling a person the name of the migration?

My own solution to this problem is/was branched_migrations, which allows you to have folders inside of db/migrate for each branch or developer or whatever. This is actually less optimal than the UTC timestamps in my opinion, but it does fix the problem of the timestamps being ugly. Maybe we could combine the approaches, having the UTC timestamps that can live in a folder? That way you could say "the migration in `jeremy` that `adds_x_to_y`". It'll cut down the project search a little. When migrating, the list could be globbed and ordered from the directories so they're kept in sync like they would be if they were in the root dir. So a folder structure like this:

    jeremy/         2008122108_add_things.rb         2008122216_add_things.rb         2008122503_add_things.rb     matt/         2008121914_add_things.rb         2008122108_add_things.rb     andy/         2008122113_add_things.rb

Would be normalized into this list...

    2008121914_add_things.rb     2008122108_add_things.rb     2008122113_add_things.rb     2008122216_add_things.rb     2008122503_add_things.rb

...and then migrated.

I don't know if that would really solve much of the problem, but it is a thought.

--Jeremy

18

Last time I checked (OK, several months ago), every migration file IS loaded for any migration..

19

Would it be possible to just move the UTC timestamps to the first line of the file? Then you could number them incrementally (if you want) and use the timestamps to break the tie when there is a conflict.

20

My thoughts exactly:

http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/631-move-migration-utc-timestamps-inside-file

It's still sans-tests unfortunately :frowning:

But give me time!