some love for migrations

rubyonrails-core
1

I have a semi-grand plan for some changes to migrations. Mostly this is motivated by the upcoming 3.0 release and wanting to get migrations in engines working.

1. Don't skip migrations with version numbers lower than "current".

I only recently tripped over this, but apparently it's been this way since 2.1 when we got timestamps masquerading as large/sparse version numbers. See ticket #4120.

rails.lighthouseapp.com/projects/8994/tickets/4120

I thought the whole point of timestamp version numbers was to be able to combine different development branches. That seems like it should run migrations older than the most recent one. I guess the alternative would be a new rake task: db:migrate:all that would run the lower-numbered versions that would otherwise be skipped.

I know there is a potential for confusion, but I don't think that confusion is any worse than now with some migrations never being run.

2. Include the contents of schema_migrations in schema.rb

The current approach for db:schema:load is to assume that all migrations in db/migrate up to the :version option have been run. This is fraught with peril. Given the issue in #1 above, it's possible to assume some skipped migrations have been run, which could really mess up your data. Since we're encouraged to rake db:schema:load to deploy, the migrations table should be populated with accurate information.

It should be pretty simple to add this info to schema.rb.

Full db dumps should probably include the full schema_migrations table contents as well.

3. Add a timestamp column to schema_migrations

I almost always want the information about when migrations were run when I'm trying to investigate an issue.

Timestamp info should not be included in a schema.rb dump, as that would both cause a lot of git thrashing and sort of be a lie when it was loaded.

4. Support for migrations in engines.

rails.lighthouseapp.com/projects/8994/tickets/2058 is pretty stale and I don't like that approach either. Since it never reached consensus I figured I'd try another approach that's worked well for Pivotal Labs' Desert system (we've been doing things that way for several years).

The basic idea is to add a method to the Migration class: migrate_engine(engine_name, target_version). You have to manually add a migration to the app's migrations, but that's nice and explicit the way we want it to be. The migration looks something like this:

     class UpgradeToMarchRelease < ActiveRecord::Migration        def self.up          migrate_engine('users', 7)          migrate_engine('blogs', 15)        end

       def self.down          migrate_engine('users', 6)          migrate_engine('blogs', 13)        end      end

Desert's approach of having a separate table to record plugin migrations had some problems, so I think the thing to do is add an "engine" column to schema_migrations to track them all in one place.

I've looked through the migration code and know just how to make this work - it's very straightforward. The engine support would work better if the changes 1 and 2 above were already done, but it's not dependent on them.

I'm starting to work on some patches for these changes, but I thought it might be worth throwing it out there for a reaction before getting too deeply into it.

2

+1 on all points.

I reviewed rails.lighthouseapp.com/projects/8994/tickets/2058, and your solutions seem to cover all bases (someone speak up if it doesn't). I especially like the part where you don't copy files to the app, I always thought that sounded crazy.

Referencing the engines' migration versions in a manually-created migration makes sense. You want to have a single "timeline" for your app's migrations, as described in this comment: https://rails.lighthouseapp.com/projects/8994/tickets/2058#ticket-2058-28 (good analysis even though it recommends the file-copy approach). Plus, manual creation can always be changed to automatic in the future with higher-level abstractions.

pre-3.0 is also the right time to add fundamental changes like this, so please post the ticket in this thread once you have a patch ready.

-- Chad

3

I have a semi-grand plan for some changes to migrations. Mostly this is motivated by the upcoming 3.0 release and wanting to get migrations in engines working.

1. Don't skip migrations with version numbers lower than "current".

I only recently tripped over this, but apparently it's been this way since 2.1 when we got timestamps masquerading as large/sparse version numbers. See ticket #4120.

rails.lighthouseapp.com/projects/8994/tickets/4120

I thought the whole point of timestamp version numbers was to be able to combine different development branches. That seems like it should run migrations older than the most recent one. I guess the alternative would be a new rake task: db:migrate:all that would run the lower-numbered versions that would otherwise be skipped.

I know there is a potential for confusion, but I don't think that confusion is any worse than now with some migrations never being run.

Agreed.

2. Include the contents of schema_migrations in schema.rb

The current approach for db:schema:load is to assume that all migrations in db/migrate up to the :version option have been run. This is fraught with peril. Given the issue in #1 above, it's possible to assume some skipped migrations have been run, which could really mess up your data. Since we're encouraged to rake db:schema:load to deploy, the migrations table should be populated with accurate information.

It should be pretty simple to add this info to schema.rb.

Full db dumps should probably include the full schema_migrations table contents as well.

Agreed.

3. Add a timestamp column to schema_migrations

I almost always want the information about when migrations were run when I'm trying to investigate an issue.

Timestamp info should not be included in a schema.rb dump, as that would both cause a lot of git thrashing and sort of be a lie when it was loaded.

This would be convenient, and nice polish, but not necessary.

4. Support for migrations in engines.

rails.lighthouseapp.com/projects/8994/tickets/2058 is pretty stale and I don't like that approach either. Since it never reached consensus I figured I'd try another approach that's worked well for Pivotal Labs' Desert system (we've been doing things that way for several years).

The basic idea is to add a method to the Migration class: migrate_engine(engine_name, target_version). You have to manually add a migration to the app's migrations, but that's nice and explicit the way we want it to be. The migration looks something like this:

class UpgradeToMarchRelease < ActiveRecord::Migration def self.up migrate_engine('users', 7) migrate_engine('blogs', 15) end

 def self\.down
   migrate\_engine\(&#39;users&#39;, 6\)
   migrate\_engine\(&#39;blogs&#39;, 13\)
 end

end

Desert's approach of having a separate table to record plugin migrations had some problems, so I think the thing to do is add an "engine" column to schema_migrations to track them all in one place.

I've looked through the migration code and know just how to make this work - it's very straightforward. The engine support would work better if the changes 1 and 2 above were already done, but it's not dependent on them.

I'm not sure about this one.

I have a plugin with migrations for its own database. I want those to run and be managed in their own universe. This works well today using our existing tools: just override the connection class method on the migration.

I have another plugin whose migrations I want mixed in with the application's. Rather than adding application migrations that compose the plugin's, I'd prefer to enlist the plugin into my app's migrations:

class Web20Portal::Application < Rails::Application   migrate :users, :blogs, :widgets end

This would simply add these plugins' migrations paths to the app's. Then the migrator proceeds as usual.

I'm starting to work on some patches for these changes, but I thought it might be worth throwing it out there for a reaction before getting too deeply into it.

Great. Thanks Josh!

jeremy

4

I'm not sure about this one.

I have a plugin with migrations for its own database. I want those to run and be managed in their own universe. This works well today using our existing tools: just override the connection class method on the migration.

You can still do that with the approach I outlined. That's a good recommendation to put in the docs.

I have another plugin whose migrations I want mixed in with the application's. Rather than adding application migrations that compose the plugin's, I'd prefer to enlist the plugin into my app's migrations:

class Web20Portal::Application < Rails::Application migrate :users, :blogs, :widgets end

This would simply add these plugins' migrations paths to the app's. Then the migrator proceeds as usual.

I hope it doesn't sound too snooty to say I've been dealing with these kinds of issues for several years and I don't think that's going to work they way you expect. Mixing in the engine migrations sounds nice in theory. In practice, I've run into situations where you really do want more manual control. Since the engine code might be out of your control, you might have to run a few of the engine migrations, one of your own in your app to adjust to those changes, then the rest of the engine ones, then the rest of your app's. And then, what if you're going to remove the engine? (perhaps replacing it with a better one) You need to migrate the engine DOWN to get rid of its stuff. It's much simpler to drop in a new migration that migrates the engine to zero and be done with it. Otherwise you have to figure out how to migrate down just the migrations for the engine without undoing the other migrations. Madness!

There's also the improbable but pathological case where a migration in an engine has the same version number or class name as one in the main app or another engine. Now that I've said that I realize that migrations in engines should be namespaced in the engine's module to avoid class name collisions.

5

You could also use Sequel's recommended approach of anonymous classes for migrations:

Class.new(ActiveRecord::Migration) do   def self.up     # ...   end   def self.down     # ...   end end

Then you don't have to worry about collisions. In addition to the engine issue, this also fixes the much more common error where people would copy a previous migration file to create a new one, and would forget to rename the migration class.

Jeremy

6

Even if a plugin/engine manages its own database, and has nothing to do with your app's database, it still operates within the context of your app and your app's timeline (you explicitly upgrade the code, you monkey patch it, etc). Otherwise, if it has nothing to do with your app, it is a separate app in itself, and should be managed independently. But since it *is* in the context of your app, it makes sense to explicitly manage the plugin's migrations within the timeline of your app's migrations. You need to have explicit control over this timeline, for all the reasons Josh mentioned in his response.

The migrations API should definitely support either approach - running an engine's migrations as a standalone app, or invoking them in the context of a parent app's migration timeline - and I hope Josh's patch takes this into account. This allows for the greatest flexibility, but we must support both.

-- Chad

7

From the point of view of someone who’s been working with engines and Pivotal’s Desert system for a few months now (on our own kind of CMS) I agree completely with Josh. Managing migrations is already a little painful in a team of developers, and when you add engines it’s even worse. The desert approach of migrating the plugin when you want has been very helpful and definatelly what we’d find most useful.