Suggestions for properly testing different Rails engine combinations?

I’ve been working to add some more end to end testing to an existing open source rails engine which provides functionality for different authentication providers (devise, authlogic, etc). One of the things I noticed is the current maintainer included 4 different “dummy” applications inside the “test” directory (one per authentication provider). I have tried multiple combinations of loading multiple Rails applications at once in order to write tests to validate the app works properly with that and I’ve come to the conclusion that there are two options:

  1. Create 4 different test suites that run in their own process.
  2. Create a dummy app that has all 4 authentication providers integrated and write tests against that single dummy app, enabling and disabling different initialization options on the fly.

My question is if I am missing some other way or if these are the only options?

How does the current maintainer run these tests? Is there a github workflow that you can look through? Have you posted this on the issues or forum for that engine?

The current maintainer is running tests that currently only test one of the providers. I have brought this up to the maintainer of the engine in an issue and provided a PR cleaning up the approach some however I figured I would ask in the broader Rails community for advice on if I was missing any other approaches that could work here. It would be super useful to be able to test with multiple dummy apps in engines but Rails really seems to hate it when you try to load multiple apps into memory at once.

The closest thing that comes to mind for something like this but not entirely what you’re asking for is the appraisals gem. Something closer to a test matrix though is probably what you’re looking for but really hard to tell without seeing more of the engine. It’s also probably possible to figure out something for this by utilizing a similar approach to the default bin/rails script within an engine or test_helper.rb. Maybe passing in an ENV var with each run or an additional script that runs with each engine, difficult to tell the best approach though without more context.

bin/rails:

ENGINE_ROOT = File.expand_path("..", __dir__)
ENGINE_PATH = File.expand_path("../lib/admin_storage/engine", __dir__)
APP_PATH = File.expand_path("../test/dummy/config/application", __dir__)

# Set up gems listed in the Gemfile.
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__)

test_helper.rb:

require_relative "../test/dummy/config/environment"
ActiveRecord::Migrator.migrations_paths = [File.expand_path("../test/dummy/db/migrate", __dir__)]
ActiveRecord::Migrator.migrations_paths << File.expand_path("../db/migrate", __dir__)

I did go ahead and integrate the apprasials gem as part of my changes, and it was super useful for testing against different gem combinations. The issue I ran into was testing against completely different application configurations and not just different versions of gems. I should’ve probably shared the link to the gem I’m talking about in the first place, wasn’t sure if I was allowed to post links or not at first: GitHub - igorkasyanchuk/any_login: Easy way to login as any user in system

Ah, gotcha. Something like this approach should probably work ruby on rails - Run multiple dummy apps in my integration tests - Stack Overflow

This is exactly what I want, the issue is Rails has a guard which prevents this from working. The issue is all of the test_helper, test_helper_1, test_helper_2, etc files get loaded and each of them loads the rails environment which calls Rails.application.initialize!. Calling initialize multiple times hits this error: rails/application.rb at d03e1d50abfcc96e66bb6903d4b12ca7bc357794 · rails/rails · GitHub and trying other things like Dummy.application.initialize! gets slightly further but hangs up on an error about modifying frozen load paths.

How are you running the tests after that?

I’m just using bundle exec rake test. I’ve setup a branch with this issue reproduced in it: GitHub - rbclark/any_login at reproducting-test-error, the relevant files are

test/integration/devise_navigation_test.rb
test/integration/authlogic_navigation_test.rb

When I run the tests on this branch I get the following error:

/Users/rbclark/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/application.rb:390:in `initialize!': Application has been already initialized. (RuntimeError)
rake aborted!
Command failed with status (1): [ruby -w -I"lib:lib:test" /Users/rbclark/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/rake_test_loader.rb "test/any_login/collection_test.rb" "test/any_login_test.rb" "test/integration/authlogic_navigation_test.rb" "test/integration/devise_navigation_test.rb" ]

If I go ahead and modify all of the load calls to be Dummy::Application.initialize!, DummyAuthlogic::Application.initialize!, etc, then my error turns into:

/Users/rbclark/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/engine.rb:588:in 
`unshift': can't modify frozen Array:

Due to rails/engine.rb at c6dbb4eb16150ead891a50b5c3feb33e1ac181f2 · rails/rails · GitHub

It’ll probably be a little difficult to get this to run as one rake task, the different test helpers will need to be run independently. Once you require a test that requires a different test helper you’re going to see that error come up. Your tests are running fine if run independently it looks like so that’s good.

Another thought that comes to mind is grouping your tests by dummy app and then having a bash script that runs each of these.

#!/usr/bin/env sh

set -eux

ruby -I test test/devise/*_test.rb
ruby -I test test/clearance/*_test.rb

Another thought I had was writing your own rake test task that forks out to a process for each type, that way you could keep it in rake if that’s your preference. You could also probably run these in parallel.

Thank you for all your suggestions on this @nickhammond, I ended up successfully implementing a test matrix following the advice you laid out: Add end to end testing for different application integrations by rbclark · Pull Request #49 · igorkasyanchuk/any_login · GitHub

@rbclark Awesome, glad you got something worked out!