plugin dependencies

rubyonrails-core
21

Ignoring anything about requiring specific versions, couldn't this be achieved by having all of the plugins added to the load path before any init.rb files are evaluated (my suggestion above)? That way the normal Ruby "require" would seem to provide everything that you describe here.

This sounds like it's probably harmless and could solve a bunch of the different issues people have been mentioning. If you wanted to take a look at this, I'd be happy to apply it.

We're definitely not going to go down the route of a massive dependency system of our own. Rubygems does this kind of thing already, and perhaps we just need to go down the plugins-as-gems approach.

Either way, I'd be open to applying a patch to add :all to config.plugins or the load order changing, but that's about it for 2.0.

22

I like the plugins-as-gems approach. I never personally saw a reason for any difference between plugins and gems, which is why I write my plugins to be supported as both. Is the only thing holding back a plugins-as-gems approach RubyGems's inability to support multiple search paths? (at least, that's what I think I had read a while back)

23

Adding something like :all is not an ideal fix as this doesn't allow any type of error reporting for the user that dependent plugins are missing or possibly being loaded incorrectly. I mentioned something similiar in my OP.

Back to why I brought this all up in the first place...

Somebody was using my plugin that I wrote to improve the in_place_editor code. The user was using edge rails. The in_place_editor code was recently moved to a plugin, so now my plugin stopped working and the user contacts me that it doesn't work in edge (which is to expected with edge code), but the user was not aware that the original rails code had been moved to an external plugin and neither was I. It turns out that simply installing the new "core plugin" fixes the problem (since the alphabetical ordering was ok in this case), but the user had no indication that this was the problem and neither did I initially. I'm sure this will become more of a common case in the future (especially with the release of rails 2).

If there was a method to report back some sort of debug information to the user informing them they need to install the additional plugin (and possibly modifying the config.plugins) property that would save many headaches I'm sure along with saving time to support users on how to fix the issue that some feedback from the plugin could easily do. I'm not saying that a full error reporting system be implemented, that could easily be accomplished with a simple print statement, but the functionality to find out if there's a dependency problem would be required.

Using :all would be better than what's currently there which is basically and all or nothing approach to loading plugins, but I think it's far from ideal.

What do you mean by "load order changing"? I've missed the details of this somewhere.

24

I release Plugin A. Joe release Plugin B which uses "your" functionality to depend on my Plugin A. One fine morning I get enormous motiviation to improve performance of my Plugin. So I take a day off at work and rewrite the entire damn thing and release it. Poor Joe has no clue what I did. So every person installing his Plugin B, will have broken functionality because noone has a goddman idea about what I did with Plugin A last night.

I see what you're saying, and I don't disagree. There's always going to be a tension between reuse and incompatibility; it's been true with UNIX shared libraries, it's true for Windows "DLL hell", I imagine it's true for managed .NET and Java applications, and it will be true in Ruby and Rails. It's why we freeze gems and Rails into our apps, and it's why 3rdRail comes with not only its own copy of Eclipse but its own copy of Java.

And that's the problem with the nothing-shared rule - just how much of your dependencies do you include? What stable, sane environment can you count on? Should plugins include Ruby? After all, 1.8.5 to 1.8.6 broke quite a few things.

But packaging everything can destabilize you too, because you won't pick up bug fixes to the core libraries. On Windows, nearly everything includes its own "gdiplus" DLL. Well, when Microsoft found a security hole in gdiplus, the world went crazy trying to release new versions of everything with an updated gdiplus. I'm pretty sure there's something somewhere on my hard drive that's vulnerable.

These aren't new problems, and we'll do well to learn from past mistakes and experiments as we carve out solutions.

If I were you, I'd refrain from personal attacks.

I didn't intend my post to be a personal attack, and I'm sorry that it made you feel attacked. I didn't make any statements about your person; it's your position that I questioned.

Perhaps it was a bit hyperbolic to suggest we all invent our linked-list libraries, but in fact 15 years ago that's exactly where we were. It took ages (and IIRC several core language improvements!) for the C++ STL to become adopted and truly standardized, because reuse, versioning and efficency (developer AND system) are huge hairy issues. I'm sure the same happened before that with stdlib, and after that with Java, and it's happening here now.

Java was born in an age where complex programs solved complex problems, and the idea of perfect portability and modular object reuse was going to save us all. It didn't.

Rails comes at the problem a different way, and perhaps as a result, Rails developers have a strong cultural bias against reuse. Maybe it's because it's often as easy to write something from scratch as it is to reuse it. Maybe it's because we're learning and reinventing so quickly that any rewrite is going to be a huge improvement. Maybe it's a rebellion against the idea of "one true way" to do something. Maybe it's because we haven't started building complex Rails apps yet; maybe it's because we *shouldn't* build any. Or maybe it's because we just haven't hit the pain point, but we're about to.

What I find really fascinating is that Perl, which comes from a very similar TIMTOWTDI perspective, has begat CPAN, which (from what I can tell) very often has One Good Library for a given function. I would love to understand more about how that evolved, and see what we can borrow.

Sorry for the novel. Executive summary: These aren't new problems. Let's learn from at old solutions.

25

I didn't intend my post to be a personal attack, and I'm sorry that it made you feel attacked. I didn't make any statements about your person; it's your position that I questioned.

Point taken. No worries :slight_smile: Thanks for clearing the air.

Sorry for the novel. Executive summary: These aren't new problems. Let's learn from at old solutions.

Exactly. That's why plugins as gems approach is better, as you can have versioning of your plugins and any other plugin can easily say "I need version X of plugingem Foo" even though the latest version of Foo is X+2.

26

Except there is no real concept of versions with plugins. Starting with simple dependency requirements of "plugin X requires plugin Y" would make me happy for now.

27

Emphasis on "for now", I hope. When your dependencies can change, it's not long before you start needing to encode temporal information in your dependency tree to keep everything working.

- Matt

28

Michael Koziarski wrote:

We're definitely not going to go down the route of a massive dependency system of our own. Rubygems does this kind of thing already, and perhaps we just need to go down the plugins-as-gems approach.

I just want to add my +1 to this comment. It seems like plugins are trying to solve all the same problems RubyGems is trying to solve. But plugins are just doing it badly while RubyGems is doing better (not perfect but much better).

In my mind plugins should be deprecated and eventually removed. Having a "gem" route and a "plugin" route is just too confusing. To me if I want to enhance Rails to have the functionality of acts_as_paranoid then I should just:

1. gem install acts_as_paranoid 2. Add "require acts_as_paranoid" in my environment.rb. If I care about binding it to a specific version of acts_as_paranoid I would instead put "require_gem 'acts_as_versioned', '0.3'"

Then I'm done. This would remove a lot of code from Rails core making it simpler but it would give us the following benefits we don't currently have:

1. Distribution - yes I know "./script/plugin install blah" but it is crappy. Requires SVN to be laid out in a non-standard way, has a flaky way of finding the package so you pretty much always have to get the SVN URL and has problems with some repositories (SSL, etc). Gem is quick, simple and works.

2. Versioning - This ensures stability. The plugin route has nothing for this other than bringing it into your own repository and then having to use third party tools like piston to allow updates when desired.

3. Dependencies - Specifying a load order is just archane. Yes it works but as I user of acts_as_paraniod I shouldn't have to care if it has a dependency. When I install acts_as_paranoid it should tell rubygems its dependencies and all I have to do is say install with all dependencies.

4. Simplicity - No more spending hours tracking to load order dependency problems. Reduction of Rails core code.

Eric

29

I like the approach, a lot. But (isn't there always a but...), isn't
this going to be a problem for people using shared hosting, or people
that just can't install gems? Unless there is an easy way to locally
install gems...

Cheers, Pascal

30

On any decent shared host you can install your own gems. If you can’t then you will be in for a world of hurt anyway, and frankly, the world of shared Rails hosting is too low of a common denominator too target. There are too many issues with memory and stability as it is, so I say we make Rails do the best thing it can assuming a VPS or dedicated box, and then let the shared hosts figure out how to make it work for $5/month.

Putting Rails’ weight behind the gem system seems like a win-win to me.

31

You can also freeze gems into the vendor directory. Now what’s the downside compared to plugins in vendor?

32

Should have thought of that. I suppose that would be workable. I’m sure someone could even come up with a script that downloads a gem and installs it directly into vendor.

That takes care of my concern :slight_smile:

Cheers,

Pascal.

33

If every plugin's lib was in the load path *before* any plugin could start playing around or monkeypatching, many situations where you might want to explicitly ensure a plugin was loaded can be made to go away.

Can you look into writing a patch for that?

For the scenarios where you do need some load order, I've requested http://dev.rubyonrails.org/ticket/9613 -- it takes just enough pain out of config.plugins to make it bearable, but still leaves enough in that people won't go crazy with dependency overload. Some one PDI this likely-just-a-few-lines patch.

34

I've come up with an initial stab at 9613

Fred

35

Pascal Belloncle wrote:

I like the approach, a lot. But (isn't there always a but...), isn't this going to be a problem for people using shared hosting, or people that just can't install gems? Unless there is an easy way to locally install gems...

I'm on Site5 which is a cheap shared host. I can install gems perfectly fine to a local directory. They have a few sitewide ones installed but I have my own installed also. The only ones I might have problems with (haven't tried) are ones that require native code.

Eric

36

Josh Susser wrote:

You can also freeze gems into the vendor directory. Now what's the downside compared to plugins in vendor?

It seems to me the versioning feature of gems solves that. Just require the gem with a specific version requested. You are now frozen on that version. Updates to the package don't affect you and you can move up to new versions just by specifying what version you want.

Eric

37

Not to beat a dead horse, but I threw together a few ideas for working towards the plugins-as-gems approach. Attached are two possible ways of going about it. They are not, by any means, complete solutions (or completely tested solutions) but offer an insight at a possibility for plugins. Both patches basically replace the current plugin loading mechanism with the gem system. This way, we get support for versioning, the gem repository, dependencies, etc.

For both patches, existing plugins would:

  1. No longer need an init.rb
  2. Follow the standard gem structure (i.e. my_plugin/lib/my_plugin.rb)
  3. Need to create a Rakefile which defines the gem specification (which some already do)

gem_always.diff - This patch assumes that everything (plugins and gems) would exist under vendor/ (a true vendor everything).

gem_optional.diff - This patch would require config.plugin_loader to be set to Rails::Plugin::GemLoader in order to use the gem system for loading plugins. This patch assumes that plugins and gems (except rails) would exist under vendor/plugins.

Essentially, plugins would now simply refer to gems installed locally in the application. This would also mean that you wouldn’t need to do anything special for loading gems which are normally stored under vendor/ or vendor/gems.

Anyway, what are everyone’s thoughts?

gem_always.diff (4.55 KB)

gem_optional.diff (2.73 KB)

38

> If every plugin's lib was in the load path *before* any plugin could > start playing around or monkeypatching, many situations where you > might want to explicitly ensure a plugin was loaded can be made to go > away.

Can you look into writing a patch for that?

Absolutely.

For the scenarios where you do need some load order, I've requested http://dev.rubyonrails.org/ticket/9613 -- it takes just enough pain out of config.plugins to make it bearable, but still leaves enough in that people won't go crazy with dependency overload. Some one PDI this likely-just-a-few-lines patch.

This is the mechanism I stuck with myself, despite requests to add some kind of dependency management via the engines plugin. This is a situation where the 80/20 (99/1?) rule can easily be applied.

39

I've just created two (independent) patches to address this issue, which you guys can now see on Trac. Both apply cleanly on edge as of 10 minutes ago, though not at the same time, of course.

The first - http://dev.rubyonrails.org/ticket/9793 - does the job with the minimal amount of change to the existing code, and would be great to apply, but still leaves me feeling like more could be done. Hence:

The second - http://dev.rubyonrails.org/ticket/9795 - which also does the job, but also refactors the plugin locator/loader relationship to be (hopefully) clearer, more testable and easier to develop in the future. It's quite a bit bigger, but that is only because it is mostly refactoring (which always inflates a diff) and testing, rather than significant new behaviour.

While I'm aware that getting feedback on this larger patch is going to be more effort on your part, once you've applied it to a clean edge checkout, browsing around the code in railties/lib/rails/ should be clear enough.

Feedback greatly appreciated!

Cheers,

40

What about looking at a script/gem install which installs the gem into the load path of a vendor/gems/ directory? This could provide the dependency requirements by leveraging what’s existing.

One thing that comes to mind which the config.plugins does that a gem won’t neccessarly do, is the load order of them.

I am strongly for the idea of using Gems instead of plugins. This has always been a wish of mine.