Plugin generator gems

Plugins seem to be the way that a lot of the interesting work in Rails
is being done at the moment. Plugins are certainly very powerful, but
the way they're installed could use some improvement. You can use
"script/plugin", which does a screen-scrape of
http://wiki.rubyonrails.org/rails/pages/Plugins then uses Subversion to
install the plugin. But increasingly plugins are being written that
depend on other plugins, which in turn depend on others etc., and this
installation method doesn't scale up very well at all.

Compare with package installation in systems such as Debian Linux,
where you just specify the package you want then dpkg handles the
details of downloading and installing the package and all its
dependencies, including updates where required. Wouldn't it be nice if
installing plugins in Rails were like this?

I think this is achievable, using a combination of generators and gems.
The system could be arranged as follows:

* The plugins are arranged as generators, with manifests and template
directories.

* These generators are packaged and distributed as gems.

They still get used as plugins in a developer's Rails project. The
difference is that the
gems take care of the installation (NOT the runtime) dependencies, and
the
generators put the files into vendor/plugins/ in the developer's
project.

After running the generator to put the plugin into a Rails project, the
developer could then run a Rake task to complete the installation. The
name of the task could be "<plugin_name>:install". This would execute
the plugin's "install.rb" file, generate any dependant plugins and
recursively run their :install tasks to get them set up too.

In summary the whole process might look something like this:

    gem install someplugin_generator
    cd myproject
    script/generate someplugin
    rake someplugin:install

This would install the "someplugin" plugin -- along with any number of
dependencies -- setting them all up in "myproject".

My idea would be to start a project that provides generator gems for
lots of useful and popular plugins that are already around at the
moment. It would involve taking the authors' original code,
re-arranging the files a bit for installation as generators, writing
the generator manifest and the gem specification files for each,
gemming them up, and releasing them as gems on RubyForge with credits
and copyright remaining with the original author.

Does this project interest anyone? Anyone got any better ideas? Want
to help?

Regards,
Dave Nelson
goldberg.240gl.org

You've given this some thought. It would be convenient for a plugin
to check and install missing dependencies. I wonder how often that
method would break because a dependent repository was relocated?

It also sounds like, your system would force all plugins to be
installed as gems, which require root access. This means that I
wouldn't be able to add a plugin to a hosted application without the
help/approval of my hosting service. It's not a problem when
developing on a box I control. And, if I use svn and deploy to the
hosted server only from my svn repository, it should get around that
issue. But, I thought I'd mention it, anyway.
-Larry

Hi Larry,

You've given this some thought. It would be convenient for a plugin
to check and install missing dependencies. I wonder how often that
method would break because a dependent repository was relocated?

Actually that's one of the problems this proposed system would *solve*.
The gems would all be on RubyForge: the default gem source. Plugins,
on the other hand, are scattered all over the 'Net.

And with gems -- unlike SVN repositories -- you only have to specify
the gem *name*, not the path to the repository.

It also sounds like, your system would force all plugins to be
installed as gems, which require root access. This means that I

Acutally, no on both counts:

* You could keep using whatever non-gem plugins you wanted. This is
not a mutually exclusive system.

* Section 3.2 of the Gem manual details how users without root access
can install gems:

http://rubygems.org/read/chapter/3

Thanks for your comments Larry.

Regards,
Dave

I’d be happy to help. I’ve been thinking for a number of months now that leveraging rubygems for installing plugins made sense. An alternative I’ve considered is just running piston at my plugin directory, to solve the issue of svn repositories going away, but that doesn’t resolve the dependency on a command-line svn for installing plugins.

I don’t know that I’m sold on using script/generate plus a rake task to install the plugin, though. Perhaps a separate command, like rapt, could be used to do that work and thus keep the number of generators from exploding.

Benjamin Curtis wrote:

I'd be happy to help. I've been thinking for a number of months now
that leveraging rubygems for installing plugins made sense. An
alternative I've considered is just running piston at my plugin
directory, to solve the issue of svn repositories going away, but
that doesn't resolve the dependency on a command-line svn for
installing plugins.

I don't know that I'm sold on using script/generate plus a rake task
to install the plugin, though. Perhaps a separate command, like
rapt, could be used to do that work and thus keep the number of
generators from exploding.

First, I want to say kudos to Dave Nelson for bringing the idea up.
It's something I've always thought about, but was too busy developing
plugins to do :slight_smile:

While the generator is certainly an interesting idea, I would
personally rather stay away from the use of generator gems and move
towards a gemspec-like implementation. I haven't thoroughly thought
this idea out, but essentially we could have something like a
pluginspec that (a) is included in the root directory of the plugin or
(b) is posted alongside an official website like Agile Web
Development's or (c) both. In this way, a plugin manager like RaPT
could read the plugin's specification and figure out what dependencies
it has.

So, to put that into code, you could have something like:

plugin_abc.pluginspec:
Plugin::Specification.new do |s|
  s.name = %q(plugin_abc)
  s.version = "1.1"
  s.date = %q(2006-10-08}
  s.summary = "Says it's abc's"
  s.email = %q(owner@email.com)
  ...
  s.add_dependency(%q(<plugin_123>, ["= 1.01"])
end

You might even be able to avoid a new "pluginspec", reuse the gemspec
(which we can already read from), and simply include that in your
plugin directory or alongside its listing on AWD. No need to
reimplement it with a new Plugin namespace, just reuse
Gem::Specification.

I think I've blogged about this before, but it appears that most
plugins can be packaged as gems without any changes in directory
structure or code. So writing a gemspec for your plugin would be one
step forward to giving your plugin the ability to be installed as a gem
OR a plugin, whichever the developer wants. But now, with a gemspec in
both, we get the added benefit of knowing dependencies and other
metadata.

There may be some additional things to think about like: If I'm
installing plugin_abc as a plugin, but one of its dependencies
(plugin_123) comes as both a plugin AND a gem, how do I know which one
the developer wants installed? That's probably something we could get
away with through just prompting the developer to select or even a
configuration option for the command to install all dependencies as
plugins or gems.

But that's my two cents. Using a mechanism that has already been
thoroughly thought out and tested, I think, is the best way to go.

What do you think?

Thanks to everyone for your comments so far.

Aaron Pfeifer wrote:

While the generator is certainly an interesting idea, I would
personally rather stay away from the use of generator gems and move
towards a gemspec-like implementation. I haven't thoroughly thought

...

But that's my two cents. Using a mechanism that has already been
thoroughly thought out and tested, I think, is the best way to go.

That's my rationale for using existing tools: I believe we can achieve
a working system without having to write any new tools. Gems are good
at installation, including dependencies. Generators are good for
getting files into any part of your project, and they can be installed
from gems. Rake is good for specifying tasks that may in turn depend
on other tasks.

Note that I'm not arguing that you should turn your plugins into
generators. Under this scheme `script/generate` is our "bitch": we're
just using it to get the plugin code from the installed gem into our
project(s). The generator bit is just part of the gem packaging. Once
the plugin is installed into your project, it is the same as the
original plugin.

There may be some additional things to think about like: If I'm
installing plugin_abc as a plugin, but one of its dependencies
(plugin_123) comes as both a plugin AND a gem, how do I know which one
the developer wants installed? That's probably something we could get

I've got that covered. What you're describing is a bug in the
*packaging* (but not a bug in the *original plugin*). The package
maintainer has packaged something without properly taking into account
its dependencies. If they had done it properly, they would have also
packaged "plugin_123" as a gem and made that available on RubyForge
too. If they had done that, the end user wouldn't even have had to
think about "plugin_123" at all: it would have been installed and set
up in the user's Rails project when they installed "plugin_abc".

Again compare with Debian: you have package maintainers and original
authors. The Debian maintainers take the original work, tweak it for
use in Debian, package it, and put it up into one of the Debian
repositories. Debian users can then get their packages from the Debian
repositories, not the original software authors. And if the users find
bugs, they report it back to the Debian project; the Debian developers
may then report those bugs upstream.

I think I've blogged about this before, but it appears that most
plugins can be packaged as gems without any changes in directory
structure or code. So writing a gemspec for your plugin would be one

Exactly. And packaging them as generator gems doesn't require any
re-arrangement either. You just have to put the whole plugin into a
templates/ directory, prepare a xxx_generator.rb file, and put all that
in your gemspec. This part of the process is so regular that it could
be scripted.

What do you think?

I'm very appreciative of your comments. With respect, I would suggest
that you take a step back and consider, "Can we do this with existing
tools?" A good argument for introducing a new system would be if you
could pinpoint one area that can't be covered with the existing tools.
Then you could say, "Dave, your proposed system doesn't do [something]
very well. We need a tool that does [something] in order to cover
that." I'd like to reduce the installation burden on the end-user,
i.e. I'd prefer not to have to say to the end user, "We've got this
fancy new plugin system, but you have to install [software] to use it."

What do *you* think? :wink:

Regards,
Dave

Dave,

This is a great idea. One of the biggest strengths of Perl has been
CPAN and the ONLY reason CPAN is a success is because it is so easy to
use. Like certain Linux package managers, you just say what you want
installed and it goes ahead and does it. Pulls everything down, hunts
for the best connection, runs the tests, tells you when the tests
fail... it's a dream.

I really think that we need a CPAN for Rails. It's something that I
have also thought about quite a lot.

I like your idea of using what's already in place. I like it to much,
that I think we need to be aware of a limitation in the discussion so
far: We are limiting ourselves to "what's already there IN RUBY". I
hate reinventing the wheel. I hate reinventing the wheel even more when
an awesomely good solution with few flaws and tried and tested over
time with great effect presently exists.

In this particular case I think that we have such a strong, working
model in CPAN, a methodology that's been refined over many years and
has proven itself. IMHO it is what made Perl. No-one else at the time
that Perl was on the rise could touch Perl because it had this central
repository of code that was available instantly and immediately useful.
You didn't have to know anything about what modules, configration or
other dependencies what you wanted required, you just said "gimme", it
did, and you used it.

So I I personally think that we need to take the "what's already there"
concept to a wider level, learn from CPAN and copy it. Those guys have
done an extremely good job. They also didn't fall into the fatal trap
of going GUI or becoming too advanced with the interface. They kept it
to the minimal, command-line interface needed to simply use the thing.
I.e. They didn't make the mistake of trying to build a whizzbang "oh
wow" set of sexy routines. They are focussed on what gets the job done
with the least fuss and as quickly as possible.

I personally don't see how we can get better than CPAN. And I think
that implementig and equivalent, with the same pattern, for Rails will
provide us with the sweet spot of funcationality that we need.

It will also bring to Rails what has worked so very well for Perl.

urbanus wrote:

Hi Brad,

This is a great idea. One of the biggest strengths of Perl has been
CPAN and the ONLY reason CPAN is a success is because it is so easy to

As an old Perl hacker I concur.

I really think that we need a CPAN for Rails. It's something that I
have also thought about quite a lot.

Well, I think that making a CPAN for Rails would be as simple as giving
Rubygems a lobotomy :wink: Rubygems is sooo much better than CPAN.
Rubygems not only installs packages and their dependencies
automatically, it also does things like builds their documentation and
gives you a trivial built-in webserver to browse their documentation
through localhost. And so much more, I could rave on and on.

Rubygems makes me cry... in a good way.

I like your idea of using what's already in place. I like it to much,
that I think we need to be aware of a limitation in the discussion so
far: We are limiting ourselves to "what's already there IN RUBY". I
hate reinventing the wheel. I hate reinventing the wheel even more when
an awesomely good solution with few flaws and tried and tested over
time with great effect presently exists.

I don't see using the features of Ruby as a limitation so much. Ruby,
Rubygems and Rails are all inter-related:

* Rails is available as a set of gems.

* That includes ActiveRecord, which isn't strictly just Rails: it can
be installed from the gem and used independently of Rails.

* Some popular Rails projects -- such as Ajax Scaffold -- are available
as generator gems.

In this particular case I think that we have such a strong, working
model in CPAN, a methodology that's been refined over many years and
has proven itself. IMHO it is what made Perl. No-one else at the time
that Perl was on the rise could touch Perl because it had this central
repository of code that was available instantly and immediately useful.
You didn't have to know anything about what modules, configration or
other dependencies what you wanted required, you just said "gimme", it
did, and you used it.

Rubygems also has a central repository: rubyforge.org.

I personally don't see how we can get better than CPAN. And I think
that implementig and equivalent, with the same pattern, for Rails will
provide us with the sweet spot of funcationality that we need.

Actually, I'm not the first person to suggest that plugins can be
packaged as generator gems. But the common reaction seems to be, "Hey,
that's a great idea! Let's build a grand unified system that will
become the most fantastic way of distributing plugins ever! Hm, sounds
like a lot of work, maybe I'll look at it later." And the idea fades
away until the next time someone comes along.

I might just have to draw a line in the sand and DO IT. That would
give everyone something concrete to evaluate.

My offer remains open to anyone who wants to help out. Please feel
free to contact me directly by e-mail. And keep your eyes open for
announcements over the coming weeks :wink:

Regards,
Dave

I agree Dave, I think that we should be doing plugin distribution with gems ultimately. And if there are shortcomings with gems (like needing a way to do non-root installations) then we need to address that shortcoming in gems.

Then as you say, we can easily gem install gem and then script/generate the plugin stub which defers most of its work to the gem.

This is exactly what we do with the MasterView project. We allow one to install as a plugin or using gems and then generate the plugin stub (which defers most of its work to the installed gem).

Using gems is nice because you can easily require dependencies, etc. Plus it is the standard way to do ruby installs. So if anything is lacking we should be building it into that and have plugins leverage gems.

So we got this all working, but it was a bit of a pain to get it all setup properly to start with. Things could be much easier. If there was an easier standardized way to accomplish this then we’d certainly support it.

Jeff