Plugin migrations - simplest approach

Hi all,

The two key features missing from the 'engines' support in Rails 2.3
are migrations and public assets; the latter isn't too hard to handle
for plugins themselves, but migrations are a bit trickier, so I'm
really keen to get something into core that is simple and consistent.

Luca Guidi has done some great work around this, but unfortunately,
it's a tricky problem. I've used the discussion here (http://
interblah.net/plugin-migrations) to clarify my own thoughts, and
hopefully others will find it useful. The two key problems are
maintaining the timeline (so that rollback is consistent), and
detecting which migrations have already been run.

I've attached what I believe to be the simplest, least-intrusive
implementation I can think of to the ticket -
http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/2058-rake-tasks-for-run-engine-migrations#ticket-2058-22

Please take a look, and let me know what you think.

James

Pivotal Labs' Desert system does a lot of what Engines does, though somewhat differently in places. It has a pretty simple system for doing migrations from plugins, and the code is open source.

http://github.com/pivotal/desert

Look in desert/lib/desert/plugin_migrations

Desert creates a parallel table for plugin migrations: plugin_schema_migrations. It's like the app's schema_migrations table, but each row has both a version and a plugin name. Migrations in plugins are accessed by putting a method call in an app migration that migrates to a particular migration version in the plugin - it's not automagic, but it works quite well.

This would probably work fine to manage migrations in Rails Engine plugins too.

That's effectively a mash of two implementations in the history of how
the engines plugin enabled migrations (the separate table to track
what's been run from where, and the app migration to ensure the
timeline is preserved), so I can definitely relate to how it works.
However, the point here is whether or not we can make this
functionality available as part of the core system, and how to do that
with a balance of simplicity and elegance. Given that vanilla plugins
can contain models, it makes sense that they should also be able to
produce the tables required by those models.

I've boiled things down to two possible implementations that satisfy
all required functional goals - one based on the existing
implementation in the engines plugin (and so Desert too), or a simpler
Rake task. After spending some time discussion what was likely to be
incorporated, I decided not to pursue any solution that modified the
migration tables (either by adding a new table or adding a 'source'
column to the default one); David just wasn't into it. Hence the
current patch, on which it'd be great if you commented specifically.

Thanks,

James

This looks good to me, much neater than having separate tables or extra columns to maintain migrations from plugins.I think it’s perfectly reasonable to expect a user of a plugin to copy the migrations that creates the models to their db/migrations directory.

The only thing I can think of is potentially hooking this into the install script so that they automatically get copied across during installation of the plugin. Or is that too much magic?

Cheers,

Tekin Suleyman

“See you at Scotland on Rails!”

Ruby on Rails Developer

+44 (0) 161 408 8868

+44 (0) 7968 355 460

http://tekin.co.uk

Unnecessarily copying a file from a plugin seems wrong. Referencing
plugin migrations from your app (as Desert does) seems right.

-- Chad

One of the reasons I prefer the copy method is that it’s much easier to examine and manage the migration history of an app if they’re all in one place. The current engines plugin behaviour is pretty much the same as desert as far as I can see and I find the indirection of having a migration reference another migration a bit of a pain. Also, the fact that the schema_migrations table remains untouched this way is a big plus.

Tekin

"Spiritually", I prefer referencing too. It just requires a mode
significant change to the way migrations work, unfortunately.

That said, I would fully support a nice clean patch that referenced
migrations, whilst preserving rollback, if that would help such a
patch get accepted.

James

At the risk of bikeshedding, I’d rather not have duplicate, semi-generated code cluttering up that directory, and likely getting checked in to version control. It was bad enough with sass…

A compromise might be to use symlinks instead, but that’s not portable.

Unlike SASS, it's actually desirable that these files exist in version
control. It's crucial that db:migrate accurately represents a timeline
describing the state of the database at each check-in. There's no way
to achieve this without putting some 'semi-generated' file in the
app's db/migrate directory; even the original engines/desert
implementation will create similar files for exactly this reason. It's
exactly not clutter :slight_smile:

I'd invite you to look at http://interblah.net/plugin-migrations for a
more in-depth 'think-through' of why this is important, but do let me
know if you're still not convinced.

Thanks for taking the time to respond,

James

At the risk of bikeshedding, I’d rather not have duplicate, semi-generated

code cluttering up that directory, and likely getting checked in to version

control. It was bad enough with sass…

A compromise might be to use symlinks instead, but that’s not portable.

Unlike SASS, it’s actually desirable that these files exist in version

control.

But they already are, within the plugin.

I’d invite you to look at http://interblah.net/plugin-migrations for a

more in-depth ‘think-through’ of why this is important, but do let me

know if you’re still not convinced.

Ah, I see your point.

I am still not convinced that it needs to be duplicated – I would still opt for a symlink, or maybe something like it:

MigrateBarInPluginFoo = SomePluginModule.some_helper_method :Foo, :Bar

Or, better:

MigratePluginFooTo20081124174854 = some_helper_method :Foo, 20081224174854

If a plugin has multiple migrations when installed, no need for them each to have their own corresponding application migration. And version control should really be about tracking the user’s own creations and changes – and this is all the user did, they very likely didn’t change anything about the content of the migration.

Of course, probably the biggest reason this disagrees with me is that I don’t really see that a plugin should conflict with existing tables. Ideally, the plugin should have namespaced models, for precisely this reason. I really have no idea, but I suspect this is how Merb Slices solves this problem.

If there’s really a potential problem of conflicting migrations, the extra column on schema_migrations would solve that, right?

> I'd invite you to look athttp://interblah.net/plugin-migrations for a
> more in-depth 'think-through' of why this is important, but do let me
> know if you're still not convinced.

Ah, I see your point.

I am still not convinced that it needs to be duplicated -- I would still opt
for a symlink,

Alas, it's not cross platform, and so while that'd be nice, we have to
ignore its appeal.

or maybe something like it:

MigrateBarInPluginFoo = SomePluginModule.some_helper_method :Foo, :Bar

Or, better:

MigratePluginFooTo20081124174854 = some_helper_method :Foo, 20081224174854

I'm not really sure what you have in mind here, but I'm happy to chalk
that up to my poor understanding - if you feel you have a good idea,
please do investigate (as they say :)). I am be no means wedded to my
patch, except that so far I feel it is the most likely-to-be-applied
solution that meets the criteria outlined on interblah.net.

And version control
should really be about tracking the user's own creations and changes -- and
this is all the user did, they very likely didn't change anything about the
content of the migration.

I'm not sure this is a very useful way of thinking about version
control. Rather than caring about what the user directly did, version
control (imho) is about preserving the state of the application at any
particular point in time, such that we can return to that state and be
confident that we have (or can) undo any changes between our current
state, and that desired previous state. Anyway, that's a bit of a
tangent, and it's probably less useful to this discussion to go into
much more detail about it.

Of course, probably the biggest reason this disagrees with me is that I
don't really see that a plugin should conflict with existing tables.
Ideally, the plugin should have namespaced models, for precisely this
reason.

Whether conflicts are likely or not is secondary (again imho) to the
fact that they *will* occur, and whatever mechanism employed should be
robust enough to handle it. It's my feeling that it won't be uncommon
for users to want to add a column here or there.

If there's really a potential problem of conflicting migrations, the extra
column on schema_migrations would solve that, right?

Solving conflicts really means ensuring that migrations are applied in
the same order every time - so they're reversible. The only benefit
the extra column would give is make it simpler to track *which*
migrations have been run; we'd still need to introduce a new migration
(or stub migration or something) to mark when in the timeline those
migrations were applied. So the extra column itself does nothing to
prevent conflicts, unfortunately.

I hope that's clear - thanks again for taking the time to consider
this.

Anyone which commit rights care to indicate if any of this discussion
is likely to be useful? :slight_smile:

Thanks again,

James

James,

thanks a ton for putting so much work into this. Looking at the long
time this has been discussed and the number of attempts to solve it
this seems to be something like the "holy grail" of engine plugins.

I've read your blog post but I don't quite get a few problems that you
point at.

In solution #1 you say that "Next to run is the application's
migration, creating a users table, which should not exist. Failure."
What's the failure here? The users table won't exist before
20080101_create_users (as long as 20071201_create_accounts does not
mess with the users table, which it shouldn't). If I don't get you
wrong we've been using a similar approach for a while and it worked
well for this scenario.

In solution #2 the misconception imho is that the app migrations are
run first and the plugin migrations run afterwards. That of course
won't work when the app wants to modify what plugins provided.
Traditionally plugins were more seen as part of the framework/
libraries, today they also might be parallel slices of an application.
In both cases the order "app first, plugins second" seems wrong to me.
If anything app migrations would need to run after plugin migrations.
Or the timelines need to be mixed.

For the "rollback problem", i.e. rolling back all migrations from a
plugin: even if plugin migration timestamps are not tracked
explicitely one can always inspect the plugin migrations directory. So
for uninstalling/rolling back a plugin a dedicated rake task could
work that looks at the plugin migrations folder and migrates the
plugin down. This obviously would clash when the app messed with the
plugin's tables and one would need to migrate the app migrations down
manually ... which imo seems ok though.

Ummm, what?

LOL. In the heat of the battle I picked the wrong email from my drafts
folder and hit "send" without even noticing it. The subject was so
similar.

Anyway. James, I wanted to put more thought into this before sending a
reply but maybe this at least helps with keeping the discussing
running :wink:

Hi Sven,

Thanks for taking the time to read my article and write your
response :slight_smile:

I've read your blog post but I don't quite get a few problems that you
point at.

In solution #1 you say that "Next to run is the application's
migration, creating a users table, which should not exist. Failure."
What's the failure here? The users table won't exist before
20080101_create_users (as long as 20071201_create_accounts does not
mess with the users table, which it shouldn't). If I don't get you
wrong we've been using a similar approach for a while and it worked
well for this scenario.

The issue, although I'll grant that my simple example doesn't
illustrate it well, is that the migrations are being applied in the
wrong order. I'll leave it up to everyone else to either flesh out a
better example, or declare the migration order is not important. I
hope that the article, as a whole, is successful at describing *why* I
believe that migration order is significant.

If not, see the example below, where an app and a plugin are dealing
with tables with the same name, instead of 'users' vs. 'accounts'.

In solution #2 the misconception imho is that the app migrations are
run first and the plugin migrations run afterwards. That of course
won't work when the app wants to modify what plugins provided.
Traditionally plugins were more seen as part of the framework/
libraries, today they also might be parallel slices of an application.
In both cases the order "app first, plugins second" seems wrong to me.
If anything app migrations would need to run after plugin migrations.
Or the timelines need to be mixed.

Here, you might be suggesting that plugin migrations should run first
(which is, I think, also what Luca's latest patch does). I don't think
this could ever work well enough to be described as a 'feature'.
Consider an example where the app has its own 'users' table, then the
developer replaces it with one from a plugin by creating an app
migration to remove his existing table, and then running the plugin
migration. Running the migrations from scratch using the above
strategy would result in two attempts to create the 'users' table (the
first from the plugin, the second from the app), which will at best
fail, and at worse silently drop the existing table.

If instead you believe that the timelines need to be mixed, I think
you'll end up with one of two solutions - either copying/
retimestamping migrations, or creating intermediate migrations in the
app timeline.

For the "rollback problem", i.e. rolling back all migrations from a
plugin: even if plugin migration timestamps are not tracked
explicitely one can always inspect the plugin migrations directory. So
for uninstalling/rolling back a plugin a dedicated rake task could
work that looks at the plugin migrations folder and migrates the
plugin down.

Yes, this is possible, and something that I spoke about a little bit
with Yehuda Katz at Scotland on Rails last week. However, I can't see
the sense in creating specific code to quite literally tear out some
tables from the database, when I believe a general mechanism is
possible without creating any conceptual overhead.

This obviously would clash when the app messed with the
plugin's tables and one would need to migrate the app migrations down
manually ... which imo seems ok though.

I'm not sure how this can be OK. In order for migrations to be a
viable mechanism, I need to be able to trust them with my production
database. If I know there's even some chance (outside of the mistakes
I make myself) that I might be forced to unpick migrations manually on
production data, I no longer feel so great about trusting any
migrations from plugins at all.

Clearly this is a big problem, conceptually at least*.

Even if we don't make any more progress here, at some point we're
going to have to produce some 'best practice' advice for people
wanting to provide migrations in their plugins, and without any
direction from the framework, that best advice is going to be
something along the lines of "use a generator"** or "copy them as part
of install.rb".

I'm not sure how best to proceed. Perhaps I should look at the
'copying stylesheets/javascripts' aspect of engines instead! :slight_smile:

James

* It gets even bigger/weirder with Rails 3, where there's no way for a
plugin to predict the ORM used by the app.

** I'll leave exploring the pitfalls of this up to the reader.