Plugin generator gems

rubyonrails-talk
1

Plugins seem to be the way that a lot of the interesting work in Rails is being done at the moment. Plugins are certainly very powerful, but the way they're installed could use some improvement. You can use "script/plugin", which does a screen-scrape of http://wiki.rubyonrails.org/rails/pages/Plugins then uses Subversion to install the plugin. But increasingly plugins are being written that depend on other plugins, which in turn depend on others etc., and this installation method doesn't scale up very well at all.

Compare with package installation in systems such as Debian Linux, where you just specify the package you want then dpkg handles the details of downloading and installing the package and all its dependencies, including updates where required. Wouldn't it be nice if installing plugins in Rails were like this?

I think this is achievable, using a combination of generators and gems. The system could be arranged as follows:

* The plugins are arranged as generators, with manifests and template directories.

* These generators are packaged and distributed as gems.

They still get used as plugins in a developer's Rails project. The difference is that the gems take care of the installation (NOT the runtime) dependencies, and the generators put the files into vendor/plugins/ in the developer's project.

After running the generator to put the plugin into a Rails project, the developer could then run a Rake task to complete the installation. The name of the task could be "<plugin_name>:install". This would execute the plugin's "install.rb" file, generate any dependant plugins and recursively run their :install tasks to get them set up too.

In summary the whole process might look something like this:

    gem install someplugin_generator     cd myproject     script/generate someplugin     rake someplugin:install

This would install the "someplugin" plugin -- along with any number of dependencies -- setting them all up in "myproject".

My idea would be to start a project that provides generator gems for lots of useful and popular plugins that are already around at the moment. It would involve taking the authors' original code, re-arranging the files a bit for installation as generators, writing the generator manifest and the gem specification files for each, gemming them up, and releasing them as gems on RubyForge with credits and copyright remaining with the original author.

Does this project interest anyone? Anyone got any better ideas? Want to help?

Regards, Dave Nelson goldberg.240gl.org

2

You've given this some thought. It would be convenient for a plugin to check and install missing dependencies. I wonder how often that method would break because a dependent repository was relocated?

It also sounds like, your system would force all plugins to be installed as gems, which require root access. This means that I wouldn't be able to add a plugin to a hosted application without the help/approval of my hosting service. It's not a problem when developing on a box I control. And, if I use svn and deploy to the hosted server only from my svn repository, it should get around that issue. But, I thought I'd mention it, anyway. -Larry

3

Hi Larry,

You've given this some thought. It would be convenient for a plugin to check and install missing dependencies. I wonder how often that method would break because a dependent repository was relocated?

Actually that's one of the problems this proposed system would *solve*. The gems would all be on RubyForge: the default gem source. Plugins, on the other hand, are scattered all over the 'Net.

And with gems -- unlike SVN repositories -- you only have to specify the gem *name*, not the path to the repository.

It also sounds like, your system would force all plugins to be installed as gems, which require root access. This means that I

Acutally, no on both counts:

* You could keep using whatever non-gem plugins you wanted. This is not a mutually exclusive system.

* Section 3.2 of the Gem manual details how users without root access can install gems:

http://rubygems.org/read/chapter/3

Thanks for your comments Larry.

Regards, Dave

4

I’d be happy to help. I’ve been thinking for a number of months now that leveraging rubygems for installing plugins made sense. An alternative I’ve considered is just running piston at my plugin directory, to solve the issue of svn repositories going away, but that doesn’t resolve the dependency on a command-line svn for installing plugins.

I don’t know that I’m sold on using script/generate plus a rake task to install the plugin, though. Perhaps a separate command, like rapt, could be used to do that work and thus keep the number of generators from exploding.

5

Benjamin Curtis wrote:

I'd be happy to help. I've been thinking for a number of months now that leveraging rubygems for installing plugins made sense. An alternative I've considered is just running piston at my plugin directory, to solve the issue of svn repositories going away, but that doesn't resolve the dependency on a command-line svn for installing plugins.

I don't know that I'm sold on using script/generate plus a rake task to install the plugin, though. Perhaps a separate command, like rapt, could be used to do that work and thus keep the number of generators from exploding.

First, I want to say kudos to Dave Nelson for bringing the idea up. It's something I've always thought about, but was too busy developing plugins to do :slight_smile:

While the generator is certainly an interesting idea, I would personally rather stay away from the use of generator gems and move towards a gemspec-like implementation. I haven't thoroughly thought this idea out, but essentially we could have something like a pluginspec that (a) is included in the root directory of the plugin or (b) is posted alongside an official website like Agile Web Development's or (c) both. In this way, a plugin manager like RaPT could read the plugin's specification and figure out what dependencies it has.

So, to put that into code, you could have something like:

plugin_abc.pluginspec: Plugin::Specification.new do |s|   s.name = %q(plugin_abc)   s.version = "1.1"   s.date = %q(2006-10-08}   s.summary = "Says it's abc's"   s.email = %q(owner@email.com)   ...   s.add_dependency(%q(<plugin_123>, ["= 1.01"]) end

You might even be able to avoid a new "pluginspec", reuse the gemspec (which we can already read from), and simply include that in your plugin directory or alongside its listing on AWD. No need to reimplement it with a new Plugin namespace, just reuse Gem::Specification.

I think I've blogged about this before, but it appears that most plugins can be packaged as gems without any changes in directory structure or code. So writing a gemspec for your plugin would be one step forward to giving your plugin the ability to be installed as a gem OR a plugin, whichever the developer wants. But now, with a gemspec in both, we get the added benefit of knowing dependencies and other metadata.

There may be some additional things to think about like: If I'm installing plugin_abc as a plugin, but one of its dependencies (plugin_123) comes as both a plugin AND a gem, how do I know which one the developer wants installed? That's probably something we could get away with through just prompting the developer to select or even a configuration option for the command to install all dependencies as plugins or gems.

But that's my two cents. Using a mechanism that has already been thoroughly thought out and tested, I think, is the best way to go.

What do you think?

6

Thanks to everyone for your comments so far.

Aaron Pfeifer wrote:

While the generator is certainly an interesting idea, I would personally rather stay away from the use of generator gems and move towards a gemspec-like implementation. I haven't thoroughly thought

...

But that's my two cents. Using a mechanism that has already been thoroughly thought out and tested, I think, is the best way to go.

That's my rationale for using existing tools: I believe we can achieve a working system without having to write any new tools. Gems are good at installation, including dependencies. Generators are good for getting files into any part of your project, and they can be installed from gems. Rake is good for specifying tasks that may in turn depend on other tasks.

Note that I'm not arguing that you should turn your plugins into generators. Under this scheme `script/generate` is our "bitch": we're just using it to get the plugin code from the installed gem into our project(s). The generator bit is just part of the gem packaging. Once the plugin is installed into your project, it is the same as the original plugin.

There may be some additional things to think about like: If I'm installing plugin_abc as a plugin, but one of its dependencies (plugin_123) comes as both a plugin AND a gem, how do I know which one the developer wants installed? That's probably something we could get

I've got that covered. What you're describing is a bug in the *packaging* (but not a bug in the *original plugin*). The package maintainer has packaged something without properly taking into account its dependencies. If they had done it properly, they would have also packaged "plugin_123" as a gem and made that available on RubyForge too. If they had done that, the end user wouldn't even have had to think about "plugin_123" at all: it would have been installed and set up in the user's Rails project when they installed "plugin_abc".

Again compare with Debian: you have package maintainers and original authors. The Debian maintainers take the original work, tweak it for use in Debian, package it, and put it up into one of the Debian repositories. Debian users can then get their packages from the Debian repositories, not the original software authors. And if the users find bugs, they report it back to the Debian project; the Debian developers may then report those bugs upstream.

I think I've blogged about this before, but it appears that most plugins can be packaged as gems without any changes in directory structure or code. So writing a gemspec for your plugin would be one

Exactly. And packaging them as generator gems doesn't require any re-arrangement either. You just have to put the whole plugin into a templates/ directory, prepare a xxx_generator.rb file, and put all that in your gemspec. This part of the process is so regular that it could be scripted.

What do you think?

I'm very appreciative of your comments. With respect, I would suggest that you take a step back and consider, "Can we do this with existing tools?" A good argument for introducing a new system would be if you could pinpoint one area that can't be covered with the existing tools. Then you could say, "Dave, your proposed system doesn't do [something] very well. We need a tool that does [something] in order to cover that." I'd like to reduce the installation burden on the end-user, i.e. I'd prefer not to have to say to the end user, "We've got this fancy new plugin system, but you have to install [software] to use it."

What do *you* think? :wink:

Regards, Dave

7

Dave,

This is a great idea. One of the biggest strengths of Perl has been CPAN and the ONLY reason CPAN is a success is because it is so easy to use. Like certain Linux package managers, you just say what you want installed and it goes ahead and does it. Pulls everything down, hunts for the best connection, runs the tests, tells you when the tests fail... it's a dream.

I really think that we need a CPAN for Rails. It's something that I have also thought about quite a lot.

I like your idea of using what's already in place. I like it to much, that I think we need to be aware of a limitation in the discussion so far: We are limiting ourselves to "what's already there IN RUBY". I hate reinventing the wheel. I hate reinventing the wheel even more when an awesomely good solution with few flaws and tried and tested over time with great effect presently exists.

In this particular case I think that we have such a strong, working model in CPAN, a methodology that's been refined over many years and has proven itself. IMHO it is what made Perl. No-one else at the time that Perl was on the rise could touch Perl because it had this central repository of code that was available instantly and immediately useful. You didn't have to know anything about what modules, configration or other dependencies what you wanted required, you just said "gimme", it did, and you used it.

So I I personally think that we need to take the "what's already there" concept to a wider level, learn from CPAN and copy it. Those guys have done an extremely good job. They also didn't fall into the fatal trap of going GUI or becoming too advanced with the interface. They kept it to the minimal, command-line interface needed to simply use the thing. I.e. They didn't make the mistake of trying to build a whizzbang "oh wow" set of sexy routines. They are focussed on what gets the job done with the least fuss and as quickly as possible.

I personally don't see how we can get better than CPAN. And I think that implementig and equivalent, with the same pattern, for Rails will provide us with the sweet spot of funcationality that we need.

It will also bring to Rails what has worked so very well for Perl.

urbanus wrote:

8

Hi Brad,

This is a great idea. One of the biggest strengths of Perl has been CPAN and the ONLY reason CPAN is a success is because it is so easy to

As an old Perl hacker I concur.

I really think that we need a CPAN for Rails. It's something that I have also thought about quite a lot.

Well, I think that making a CPAN for Rails would be as simple as giving Rubygems a lobotomy :wink: Rubygems is sooo much better than CPAN. Rubygems not only installs packages and their dependencies automatically, it also does things like builds their documentation and gives you a trivial built-in webserver to browse their documentation through localhost. And so much more, I could rave on and on.

Rubygems makes me cry... in a good way.

I like your idea of using what's already in place. I like it to much, that I think we need to be aware of a limitation in the discussion so far: We are limiting ourselves to "what's already there IN RUBY". I hate reinventing the wheel. I hate reinventing the wheel even more when an awesomely good solution with few flaws and tried and tested over time with great effect presently exists.

I don't see using the features of Ruby as a limitation so much. Ruby, Rubygems and Rails are all inter-related:

* Rails is available as a set of gems.

* That includes ActiveRecord, which isn't strictly just Rails: it can be installed from the gem and used independently of Rails.

* Some popular Rails projects -- such as Ajax Scaffold -- are available as generator gems.

In this particular case I think that we have such a strong, working model in CPAN, a methodology that's been refined over many years and has proven itself. IMHO it is what made Perl. No-one else at the time that Perl was on the rise could touch Perl because it had this central repository of code that was available instantly and immediately useful. You didn't have to know anything about what modules, configration or other dependencies what you wanted required, you just said "gimme", it did, and you used it.

Rubygems also has a central repository: rubyforge.org.

I personally don't see how we can get better than CPAN. And I think that implementig and equivalent, with the same pattern, for Rails will provide us with the sweet spot of funcationality that we need.

Actually, I'm not the first person to suggest that plugins can be packaged as generator gems. But the common reaction seems to be, "Hey, that's a great idea! Let's build a grand unified system that will become the most fantastic way of distributing plugins ever! Hm, sounds like a lot of work, maybe I'll look at it later." And the idea fades away until the next time someone comes along.

I might just have to draw a line in the sand and DO IT. That would give everyone something concrete to evaluate.

My offer remains open to anyone who wants to help out. Please feel free to contact me directly by e-mail. And keep your eyes open for announcements over the coming weeks :wink:

Regards, Dave

9

I agree Dave, I think that we should be doing plugin distribution with gems ultimately. And if there are shortcomings with gems (like needing a way to do non-root installations) then we need to address that shortcoming in gems.

Then as you say, we can easily gem install gem and then script/generate the plugin stub which defers most of its work to the gem.

This is exactly what we do with the MasterView project. We allow one to install as a plugin or using gems and then generate the plugin stub (which defers most of its work to the installed gem).

Using gems is nice because you can easily require dependencies, etc. Plus it is the standard way to do ruby installs. So if anything is lacking we should be building it into that and have plugins leverage gems.

So we got this all working, but it was a bit of a pain to get it all setup properly to start with. Things could be much easier. If there was an easier standardized way to accomplish this then we’d certainly support it.

Jeff