Plugin migrations - simplest approach

rubyonrails-core
1

Hi all,

The two key features missing from the 'engines' support in Rails 2.3 are migrations and public assets; the latter isn't too hard to handle for plugins themselves, but migrations are a bit trickier, so I'm really keen to get something into core that is simple and consistent.

Luca Guidi has done some great work around this, but unfortunately, it's a tricky problem. I've used the discussion here (http:// interblah.net/plugin-migrations) to clarify my own thoughts, and hopefully others will find it useful. The two key problems are maintaining the timeline (so that rollback is consistent), and detecting which migrations have already been run.

I've attached what I believe to be the simplest, least-intrusive implementation I can think of to the ticket - http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/2058-rake-tasks-for-run-engine-migrations#ticket-2058-22

Please take a look, and let me know what you think.

James

2

Pivotal Labs' Desert system does a lot of what Engines does, though somewhat differently in places. It has a pretty simple system for doing migrations from plugins, and the code is open source.

http://github.com/pivotal/desert

Look in desert/lib/desert/plugin_migrations

Desert creates a parallel table for plugin migrations: plugin_schema_migrations. It's like the app's schema_migrations table, but each row has both a version and a plugin name. Migrations in plugins are accessed by putting a method call in an app migration that migrates to a particular migration version in the plugin - it's not automagic, but it works quite well.

This would probably work fine to manage migrations in Rails Engine plugins too.

3

That's effectively a mash of two implementations in the history of how the engines plugin enabled migrations (the separate table to track what's been run from where, and the app migration to ensure the timeline is preserved), so I can definitely relate to how it works. However, the point here is whether or not we can make this functionality available as part of the core system, and how to do that with a balance of simplicity and elegance. Given that vanilla plugins can contain models, it makes sense that they should also be able to produce the tables required by those models.

I've boiled things down to two possible implementations that satisfy all required functional goals - one based on the existing implementation in the engines plugin (and so Desert too), or a simpler Rake task. After spending some time discussion what was likely to be incorporated, I decided not to pursue any solution that modified the migration tables (either by adding a new table or adding a 'source' column to the default one); David just wasn't into it. Hence the current patch, on which it'd be great if you commented specifically.

Thanks,

James

4

This looks good to me, much neater than having separate tables or extra columns to maintain migrations from plugins.I think it’s perfectly reasonable to expect a user of a plugin to copy the migrations that creates the models to their db/migrations directory.

The only thing I can think of is potentially hooking this into the install script so that they automatically get copied across during installation of the plugin. Or is that too much magic?

Cheers,

Tekin Suleyman

“See you at Scotland on Rails!”

Ruby on Rails Developer

+44 (0) 161 408 8868

+44 (0) 7968 355 460

http://tekin.co.uk

5

Unnecessarily copying a file from a plugin seems wrong. Referencing plugin migrations from your app (as Desert does) seems right.

-- Chad

6

One of the reasons I prefer the copy method is that it’s much easier to examine and manage the migration history of an app if they’re all in one place. The current engines plugin behaviour is pretty much the same as desert as far as I can see and I find the indirection of having a migration reference another migration a bit of a pain. Also, the fact that the schema_migrations table remains untouched this way is a big plus.

Tekin

7

"Spiritually", I prefer referencing too. It just requires a mode significant change to the way migrations work, unfortunately.

That said, I would fully support a nice clean patch that referenced migrations, whilst preserving rollback, if that would help such a patch get accepted.

James

8

At the risk of bikeshedding, I’d rather not have duplicate, semi-generated code cluttering up that directory, and likely getting checked in to version control. It was bad enough with sass…

A compromise might be to use symlinks instead, but that’s not portable.

9

Unlike SASS, it's actually desirable that these files exist in version control. It's crucial that db:migrate accurately represents a timeline describing the state of the database at each check-in. There's no way to achieve this without putting some 'semi-generated' file in the app's db/migrate directory; even the original engines/desert implementation will create similar files for exactly this reason. It's exactly not clutter :slight_smile:

I'd invite you to look at interblah.net - Plugin Migrations for a more in-depth 'think-through' of why this is important, but do let me know if you're still not convinced.

Thanks for taking the time to respond,

James

10

At the risk of bikeshedding, I’d rather not have duplicate, semi-generated

code cluttering up that directory, and likely getting checked in to version

control. It was bad enough with sass…

A compromise might be to use symlinks instead, but that’s not portable.

Unlike SASS, it’s actually desirable that these files exist in version

control.

But they already are, within the plugin.

I’d invite you to look at http://interblah.net/plugin-migrations for a

more in-depth ‘think-through’ of why this is important, but do let me

know if you’re still not convinced.

Ah, I see your point.

I am still not convinced that it needs to be duplicated – I would still opt for a symlink, or maybe something like it:

MigrateBarInPluginFoo = SomePluginModule.some_helper_method :Foo, :Bar

Or, better:

MigratePluginFooTo20081124174854 = some_helper_method :Foo, 20081224174854

If a plugin has multiple migrations when installed, no need for them each to have their own corresponding application migration. And version control should really be about tracking the user’s own creations and changes – and this is all the user did, they very likely didn’t change anything about the content of the migration.

Of course, probably the biggest reason this disagrees with me is that I don’t really see that a plugin should conflict with existing tables. Ideally, the plugin should have namespaced models, for precisely this reason. I really have no idea, but I suspect this is how Merb Slices solves this problem.

If there’s really a potential problem of conflicting migrations, the extra column on schema_migrations would solve that, right?

11

> I'd invite you to look athttp://interblah.net/plugin-migrations for a > more in-depth 'think-through' of why this is important, but do let me > know if you're still not convinced.

Ah, I see your point.

I am still not convinced that it needs to be duplicated -- I would still opt for a symlink,

Alas, it's not cross platform, and so while that'd be nice, we have to ignore its appeal.

or maybe something like it:

MigrateBarInPluginFoo = SomePluginModule.some_helper_method :Foo, :Bar

Or, better:

MigratePluginFooTo20081124174854 = some_helper_method :Foo, 20081224174854

I'm not really sure what you have in mind here, but I'm happy to chalk that up to my poor understanding - if you feel you have a good idea, please do investigate (as they say :)). I am be no means wedded to my patch, except that so far I feel it is the most likely-to-be-applied solution that meets the criteria outlined on interblah.net.

And version control should really be about tracking the user's own creations and changes -- and this is all the user did, they very likely didn't change anything about the content of the migration.

I'm not sure this is a very useful way of thinking about version control. Rather than caring about what the user directly did, version control (imho) is about preserving the state of the application at any particular point in time, such that we can return to that state and be confident that we have (or can) undo any changes between our current state, and that desired previous state. Anyway, that's a bit of a tangent, and it's probably less useful to this discussion to go into much more detail about it.

Of course, probably the biggest reason this disagrees with me is that I don't really see that a plugin should conflict with existing tables. Ideally, the plugin should have namespaced models, for precisely this reason.

Whether conflicts are likely or not is secondary (again imho) to the fact that they *will* occur, and whatever mechanism employed should be robust enough to handle it. It's my feeling that it won't be uncommon for users to want to add a column here or there.

If there's really a potential problem of conflicting migrations, the extra column on schema_migrations would solve that, right?

Solving conflicts really means ensuring that migrations are applied in the same order every time - so they're reversible. The only benefit the extra column would give is make it simpler to track *which* migrations have been run; we'd still need to introduce a new migration (or stub migration or something) to mark when in the timeline those migrations were applied. So the extra column itself does nothing to prevent conflicts, unfortunately.

I hope that's clear - thanks again for taking the time to consider this.

Anyone which commit rights care to indicate if any of this discussion is likely to be useful? :slight_smile:

Thanks again,

James

12

James,

thanks a ton for putting so much work into this. Looking at the long
time this has been discussed and the number of attempts to solve it
this seems to be something like the "holy grail" of engine plugins.

I've read your blog post but I don't quite get a few problems that you
point at.

In solution #1 you say that "Next to run is the application's
migration, creating a users table, which should not exist. Failure."
What's the failure here? The users table won't exist before
20080101_create_users (as long as 20071201_create_accounts does not
mess with the users table, which it shouldn't). If I don't get you
wrong we've been using a similar approach for a while and it worked
well for this scenario.

In solution #2 the misconception imho is that the app migrations are
run first and the plugin migrations run afterwards. That of course
won't work when the app wants to modify what plugins provided.
Traditionally plugins were more seen as part of the framework/ libraries, today they also might be parallel slices of an application.
In both cases the order "app first, plugins second" seems wrong to me.
If anything app migrations would need to run after plugin migrations.
Or the timelines need to be mixed.

For the "rollback problem", i.e. rolling back all migrations from a
plugin: even if plugin migration timestamps are not tracked
explicitely one can always inspect the plugin migrations directory. So
for uninstalling/rolling back a plugin a dedicated rake task could
work that looks at the plugin migrations folder and migrates the
plugin down. This obviously would clash when the app messed with the
plugin's tables and one would need to migrate the app migrations down
manually ... which imo seems ok though.

13

Ummm, what?

LOL. In the heat of the battle I picked the wrong email from my drafts
folder and hit "send" without even noticing it. The subject was so
similar.

Anyway. James, I wanted to put more thought into this before sending a
reply but maybe this at least helps with keeping the discussing
running :wink:

14

Hi Sven,

Thanks for taking the time to read my article and write your response :slight_smile:

I've read your blog post but I don't quite get a few problems that you
point at.

In solution #1 you say that "Next to run is the application's
migration, creating a users table, which should not exist. Failure."
What's the failure here? The users table won't exist before
20080101_create_users (as long as 20071201_create_accounts does not
mess with the users table, which it shouldn't). If I don't get you
wrong we've been using a similar approach for a while and it worked
well for this scenario.

The issue, although I'll grant that my simple example doesn't illustrate it well, is that the migrations are being applied in the wrong order. I'll leave it up to everyone else to either flesh out a better example, or declare the migration order is not important. I hope that the article, as a whole, is successful at describing *why* I believe that migration order is significant.

If not, see the example below, where an app and a plugin are dealing with tables with the same name, instead of 'users' vs. 'accounts'.

In solution #2 the misconception imho is that the app migrations are
run first and the plugin migrations run afterwards. That of course
won't work when the app wants to modify what plugins provided.
Traditionally plugins were more seen as part of the framework/ libraries, today they also might be parallel slices of an application.
In both cases the order "app first, plugins second" seems wrong to me.
If anything app migrations would need to run after plugin migrations.
Or the timelines need to be mixed.

Here, you might be suggesting that plugin migrations should run first (which is, I think, also what Luca's latest patch does). I don't think this could ever work well enough to be described as a 'feature'. Consider an example where the app has its own 'users' table, then the developer replaces it with one from a plugin by creating an app migration to remove his existing table, and then running the plugin migration. Running the migrations from scratch using the above strategy would result in two attempts to create the 'users' table (the first from the plugin, the second from the app), which will at best fail, and at worse silently drop the existing table.

If instead you believe that the timelines need to be mixed, I think you'll end up with one of two solutions - either copying/ retimestamping migrations, or creating intermediate migrations in the app timeline.

For the "rollback problem", i.e. rolling back all migrations from a
plugin: even if plugin migration timestamps are not tracked
explicitely one can always inspect the plugin migrations directory. So
for uninstalling/rolling back a plugin a dedicated rake task could
work that looks at the plugin migrations folder and migrates the
plugin down.

Yes, this is possible, and something that I spoke about a little bit with Yehuda Katz at Scotland on Rails last week. However, I can't see the sense in creating specific code to quite literally tear out some tables from the database, when I believe a general mechanism is possible without creating any conceptual overhead.

This obviously would clash when the app messed with the
plugin's tables and one would need to migrate the app migrations down
manually ... which imo seems ok though.

I'm not sure how this can be OK. In order for migrations to be a viable mechanism, I need to be able to trust them with my production database. If I know there's even some chance (outside of the mistakes I make myself) that I might be forced to unpick migrations manually on production data, I no longer feel so great about trusting any migrations from plugins at all.

Clearly this is a big problem, conceptually at least*.

Even if we don't make any more progress here, at some point we're going to have to produce some 'best practice' advice for people wanting to provide migrations in their plugins, and without any direction from the framework, that best advice is going to be something along the lines of "use a generator"** or "copy them as part of install.rb".

I'm not sure how best to proceed. Perhaps I should look at the 'copying stylesheets/javascripts' aspect of engines instead! :slight_smile:

James

* It gets even bigger/weirder with Rails 3, where there's no way for a plugin to predict the ORM used by the app.

** I'll leave exploring the pitfalls of this up to the reader.