Suggested strategies for testing a gem against Rails 3.x and Rails 4?

rubyonrails-core
1

I’ve seen a few examples of dummy Rails apps (for testing, so they live under test or spec dirs, typically) for use with the Appraisals gem that supposedly work with both Rails 3.x and Rails 4, but they seem hackish and not fully functional. It is somewhat expected, as it is a stripped down Frankenstein monster that is trying to be compatible with various versions of Rails 3 as well as Rails 4.

I’ve referred to projects that attempt to do this sort of testing (as of late March 2013) like less-rails and ember-rails, but this way to test with various version of Rails doesn’t seem very clean, and it is non-trivial to try to debug a non-standard Rails application, especially in a beta version of Rails.

It would be great to have a cleaner way to test that allows you to have a full Rails application for each version of Rails to test with that through some magic is not that difficult to setup or maintain and don’t require non-standard path hacks in places, etc.

What are the available strategies for testing gems with various versions of Rails (including at least latest Rails 3.1.x, 3.2.x, and 4.0.0.beta1), and what are the pros and cons of each?

I’ve also asked this question at:

Thanks!

2

Note: the reason I bring this up on the core list is that I think it would help people upgrade to Rails 4 more quickly if it were easier or clearer as to how to test gems against (maybe) 3.1.x, 3.2.x, and 4.x, at least until mostly everyone switches to 4.

3

Why not test it in travis? Or some other kind of CI solution.

4

I test minitest-spec-rails against 3.0, 3.1, 3,2 and 4.0 using a mix of appraisal and dummy_app that minimally configures itself depending which rails version it is testing against. Some links:

https://github.com/metaskills/minitest-spec-rails

https://github.com/metaskills/minitest-spec-rails/blob/master/test/dummy_app/init.rb

  • HTH,

Ken

5

Ken,

It definitely helps to have more examples, and what you did in minitest-spec-rails is certainly very clean-looking!

I looked at that and the stuff you did in less-rails which was similar: https://github.com/metaskills/less-rails/blob/master/test/dummy_app/init.rb

as well as what had been done in ember-rails: https://github.com/emberjs/ember-rails/tree/master/test/dummy

6

To expand on the idea some, basically the strategy might be to:

  1. Have a list of the versions of Rails to test against (like Appraisals).

  2. For each version of Rails, generate a new Rails application called “Dummy” and put it in its own directory, e.g. test/appraising/rails-4.0.0.beta1/pristine.

  3. You would specify an overlay directory for one or more versions of Rails in a config file and put the files to overlay in the configured directory, e.g. test/appraising/overlays/rails-4.x/

  4. You would specify paths to delete as part of the overlay in that config file also.

7

And that may be fine. Thanks to changes for engines in 3.0 and upward, this is only a few lines of code. If you wanted to share models, etc, each directory can be symlinked. So you could have test/dummy_app_3 and test/dummy_app_4. Each with their own init.rb, etc. It just all depends on what you need, so the strategy will vary, and I think that is OK. From where I sit, Rails has done the work we need.

  • Ken
8

I’ve not seen that many symlinks checked into Github repos. I also don’t think that using them is all that clear because you may assume you are only changing a file in version X when that is really a symlink to version Y, and unless it is a hard link, then if you delete version Y, you just lost part of version X.

Overlays (or something similar) should allow for the most flexibility and longevity when it comes to testing a gem with various versions of Rails. Rails may have what is needed for each currently supported stable version when viewed as a silo of that version only, and provides some degree of ability to use different configuration for different Rails versions, but I am not convinced that conditional-laden config files on their own are the answer.

That said, I think that the approach you’ve taken (without symlinks afaik) is the best I’ve seen so far. I just wish that I knew that what I was testing was really something that a user could easily reproduce (i.e. it is working with known changes from a default Rails application that could be generated).

I don’t know that Rails itself needs to do anything other than what it does already to make this happen, but I brought it up here because it is something that affects anyone that wants to maintain a gem for Rails, and some of those gems might be part of Rails or used directly by Rails.

9

I’ve been meaning to discuss this topic more, as I’ve been doing it for a bunch of my gems lately. I have two that do this:

Draper: https://github.com/drapergem/draper

LocaleSetter: https://github.com/jcasimir/locale_setter/

Basically, I embed an entire Rails application into the gem, and then run it against multiple versions of Rails on travis via env vars.

10

Steve,

Thanks for sharing those! That’s a cool way to not have to use appraisal and to be able to use a single Gemfile instead.

I guess travis-ci starts up a new vm each time, so not managing gemsets or different bundles is not a problem unless you were to set a different env var value for for RAILS_VERSION at command-line, and did a bundle install, in which case you might end up with a newer version of a gem than you intended to use? Maybe you were going to manage gems in different bundles (vs. using rbenv-gemset or rvm gemsets)? It is good with the appraisal gem that it manages these separately so you can somewhat run in different environments locally before having to commit to test with travis. Did I miss something? I just did a quick look at it. The ability to test against Rails master looks cool.

Thanks again, Gary

11

Yes, there’s a theoretical possibility that it could happen. That said, I don’t ever run these tests by hand, they’re really just so that Travis can detect this kind of thing. If it was a problem, you can just blow away the lock and re-bundle.