Hello,
Summary
After watching the @eileencodes RailsConf keynote, I decided to resurrect an idea that Rails engines were the key to easy Rails scaling in larger companies since it promotes separation while being easier than SOA. Part of that separation is the need to have engines that can connect to their own database, promoting the portability of the engine.
Dan Manges articulates the gist of the engines only idea pretty well.
This is my original, functioning project from 2 years ago.
Current progress towards this goal.
Details of how an engine will work
Start with a separate, but otherwise normal, database.yml stored in the engine. The engine’s namespaced ApplicationRecord will establish a connection using that database.yml. Table prefix is also dropped since it is in its own database.
I think there is probably a better, less verbose way to have runtime stuff happen, but just haven’t figured it out yet.
In addition, we should expose new rake tasks, something like this from my original project.
Details of how the plugin generator will work
Right now I just added a separate_database flag so the full command would be:
be rails plugin --full --mountable --separate_database bar_service
``
Thanks for your time,
Kelly Wolf Stannard
It’s an interesting idea, but I’m not sure that a lot of people will need or even want to isolate engines into their own databases — in most cases, I’d expect people to want to link their application models to engine models (e.g. users to posts), which means they’d naturally sit in the same database.
I think you’ll need to make a very clear use-case for separating databases on a per-engine basis to get traction.
Thank you for your response!
You
are right in that single database is the easiest and most natural form of application. I think people reach for services way too early so don’t
expect an impassioned defense of SOA from me. That said, I think that multiple database is a feature that will have buy in for much the same reason SOA has buy in. Some people can’t handle all their requests with a
single database. For reference, this is Eileen’s keynote:
https://youtu.be/8evXWvM4oXM?t=1130
Basically, think
of this as SOA, but the applications are engines, and instead of communicating via HTTP, the applications communicate via method calls.
I second Kelly’s response. Even one of my current applications is having requirement of interacting with multiple database for different purposes.
The product(http://fanmojo.in) is itself on micro-service based architecture, where a few parts are developed as individual micro-services in Rails. Now, one Rails micro-service itself needs to interact with multiple databases for different purposes.
Say, Admin-Panel is micro-service in Rails, now it needs to interact in the following ways:
- MySQL for User-base type of data.
- DynamoDB/MongoDB for NoSQL type data.
- Redis for current state of Live/Volatile type of data.
So, a namespace based models could have easily separated out things.
Even while migrating our data via application interface(had to use application interface due to certain schema changes), we had do some little hack-type tricks to make it done(though it was easy stuff too).
For what it’s worth, we’d be interested in per-engine database configuration as well.
We have an (admittedly unusually complicated) architecture in which it would be great for us to be able to package all of our authentication-related code into an engine so that each of our applications can have it isolated from the rest of their code. We’re also using Oracle and storing our authentication-related code in a separate database schema, which means the per-engine isolation makes even more sense.
This is an interesting idea, thanks for sharing it. We use multiple databases in our Rails app at Root using GitHub - ankane/multiverse: Multiple databases for Rails
Having the database.yml inside the engine directory itself might be problematic. For us at Root, that’d work well since all of our engines are inside our project. How would this work for engines loaded via gems? It’d be hard for somebody to include an engine in their app via a gem and then modify the database.yml. I wonder if it’d be better to have the top-level config/database.yml define the credentials for the engine database somehow. An engine-owned file could be built to use environment variables, but we’ve also defined other database connection parameters in the yaml file, like disabling prepared statements and setting statement timeouts and the connection pool size.
Figuring out how to manage the connections should be solvable. After that though, Rails will still need to solve all of the migration issues mentioned in the @eileencodes RailsConf keynote. From my perspective, that’s the bigger challenge. Rails’ current approach to migrations assumes one database, and that’s why gems like multiverse are necessary to use multiple. Using establish_connection is pretty elegant, but the migrations code is too tightly coupled to one database connection set on ActiveRecord::Base. From my perspective it’d be nice to resolve that first. Ideally, all of the migration code should be able to work by passing in a database connection, rather than needing to modify the global state set on ActiveRecord::Base.
Dan
Thank you for joining in Dan.
Most likely, we need namespaced ENV variables. If you think of each engine as a separate application the most obvious thing would be to control each connection like you would a normal application’s connection, except you just namespace it.
I linked to my database migration solution in the OP. Since the engines are basically separate applications, the migrations will be run separately and possibly even concurrently.
Could this be address with ConnectionHandler? I imagine models in each engine could call with “establish_connection :my_engine” (or inherit from a base class a’la ApplicationRecord) and then database.yml could provide per-engine connection settings. If an engine needs to use the “default” database then it’d just inherit the settings for production.
Best regards
Greg Navis
Yes, I didn’t do the ApplicationRecord trick in my original project, but that plus connection handler seems like the best way to handle the connection on a per-engine basis.
