Easier modular/component architecture or "Make the Monolith Majestic"

I’d like to see Rails adopt some sort of modular architecture/component/domain driven patterns. DHH champions the majestic monolith, but nothing about Rails inherently steers the developers towards anything “majestic”. Breaking apps up by engines comes with many questions & challenges. Get too crazy with namespaces and you’ll run into more problems … for example index names can quickly become too long for postgres.

Cheers.

4 Likes

Can you speak more to the challenges you’ve experienced in breaking apps up along these lines? I have my own experiences here but I’m more interested in yours.

1 Like

Breaking apps up by engines comes with many questions & challenges

That’s true, engines are not so easy to use for component-based architecture; though that’s possible. We use them in many large Rails monoliths (and I shared some tips at the latest RailsConf Between monoliths and microservices).

I don’t think that could be something easier than engines but we can try to smooth the rough edges.

Could you provide a bit more context on the engines problems which stop you from using them?

1 Like

There are many links/articles/resource which cover the decisions people had to make (I’ll include a few below) … and then just a few days ago I watched your video actually!!

Everyone who has implemented their own component/domain/modularity solutions has faced similar challenges and made slightly different decision in terms of making it happen.

Your video covered many customizations you implemented to deal with things like gems and dependencies, migrations …

Depending on what works best - maybe this can’t be a feature on legacy rails apps but could be on new projects - either new projects can be initialized with module support, or existing projects can add modules with a rails command

rails module:add invoicing

  • … creates module directory/directories skeleton
  • … sets up gem management to be easy to use (ideally just modifying a module specific Gemfile)
  • … sets up testing/test runner
  • … the master rails app should be configured to work with this module … from running migrations from the master dir to potentially running tests across all moduels.

Take inspiration from Hanami. As you mentioned in your talk … Shopify, Gusto, and the other big players using Rails … look at the commonality across them all. Then put it “on rails” with a single command to initialize a module.

Take it further and provides base models/structure/developer tools to encourage best practices to succeeding with modular design - such as avoiding bi-directional dependencies, dealing with crossing boundaries that are explicit and limited.

http://tech.taskrabbit.com/blog/2014/02/11/rails-4-engines/

Sorry I didn’t have much time to edit this post for clarity.

3 Likes

I like this idea :+1: Currently, we use generators for that; and we have a CLI to run engines commands from the master dir.

I’d say we have a PoC :slightly_smiling_face:

Speaking of integrating something like this into Rails, I’d say that we should start with an external gem first; like Rails API, for example. Get some feedback and adoption, and, maybe, merge it into Rails someday (or at least add it to rails GitHub organization and make an official add-on, like, for example, Webpacker).

4 Likes

Could be great to hear what’s the problem with engines and improve on that before we come up with some new implementation, they’re supposed to solve just that.

In my experience, people try to use engines to solve two different problems:

  1. packaging code containing controllers (and sometimes views) for distribution in gem form.
  2. splitting up a monolith.

I have found the former case much better supported.

For me, the biggest issue I’ve found when trying to use engines to split up a monolith is figuring out how to access shared view & controller code within the host app without giving in and treating the controller & view layers as a giant, non-modular, global space.

It’s possible to get past this but IME it requires getting real comfy with the ActionController internals.

3 Likes

Perhaps this is another source of inspiration Contexts — Phoenix v1.6.12.

It seems like the newer frameworks are trying to give developers tools to develop modular monoliths.

1 Like

I would mention Django’s apps Applications | Django documentation | Django . Django encourages modularity by default. Does your app have a part that is a forum? You add a forum app. Does it have a part that handles videos? Add a videos app. Each comes with it’s own routes / views / controllers / models. Nothing really new about this, must have been in Django for ages.

This approach has it’s own pros and cons, I don’t think it’s right for most projects by default. BUT, once you do want to go modular I felt the way Django solved it was better than what Rails engines can do.

3 Likes

hey @Vlad_Dem I enjoyed the talk. I have to say though you guys ran into quite a lot of problems. Do you think your use case was different than what most people do or is this expected to happen to a bunch of people? It’s quite discouraging to try engines when you see all the potential pit falls. Nothing was really insurmountable but all of it together looks like too much work.

I haven’t seen a lot of engines usage in the wild. Mostly they were used to extract some functionality into a library, not to build the whole application with only engines. In that case, you don’t have most of these problems, I think.

Thus, I’d say component-based applications are pretty rare.

Also, many problems I discussed in the talk could not be considered as problems by other developers at all (e.g., syncing dependencies—almost no one cares :slightly_smiling_face:).

1 Like

For me, the biggest issue I’ve found when trying to use engines to split up a monolith is figuring out how to access shared view & controller code within the host app without giving in and treating the controller & view layers as a giant, non-modular, global space.

Could it be the more general difficulty of splitting your domain to bounded components Or do you think this is something Rails engines should be able to take care of better?

I’m curious - what problems are you guys trying to solve by adopting a modular structure?

Oh, definitely an engines thing. All of the times I’ve tried to use engines to split something off, it’s been a well-bounded area of functionality. The big issue is always managing view and controller code in a way that splits the difference between “entirely isolated” and "giant cluttered global namespace with different engines stomping on each other. Migration management between the host app and the engine is the only other real issue I’d mentioned and that’s mostly annoying, not difficult.

The one time I would consider an unqualified success was actually the time that there was the most legacy entanglement to detangle. But it was also a case where the only things I actually needed to extract were in the model layer. So I could totally bypass the issues around views & controllers and focus on the actual business-related engineering problems.

2 Likes

At this point, when I want to enforce modularity within a codebase I don’t bother with engines; I just make a bunch of namespaces. It’s a little annoying to always be going models/auth/user, controllers/auth/users_controller and a little annoying to enforce social rules around what ways to breach module boundaries are and aren’t allowed. But it’s lightweight and doesn’t lead to fighting with Rails all the time the way that using engines for modularity does. I wish that there were a bit more support for this style – in particular, I’d love it if I could do app/auth/models instead of app/models/auth – but it’s pretty workable for the Rails we have. It’s certainly much easier than trying to leverage a technology designed for gem packaging (engines) in order to solve a very different problem (complexity management within a monolith).

6 Likes

Agreed that namespaces are a good solution. I’ve had success with extracting gem-based modules by running bundle gem. With namespaces in place gems tend to start building themselves even before they’ve been extracted.

I’ve also found the book Component-Based Rails Applications as a thorough guide to decomposing gems or engines.

Speaking for myself, I want to extract some common code that I use in multiple applications for, eg., authentication, password resets, settings pages, common styles, etc.

For many years past I was too lazy to learn about gems & engines and solved the problem by copy-pasting (which worked about as well as you might expect).

I finally resolved to learn how to extract a gem/engine this year and found it quite confusing and complicated.

I meant to add that I am also transitioning from mostly HAML to mostly React and haven’t even begun to think about how I will modularize my client-side code. I’m still copy-pasting login screens, for example.

This is 100% spot on with my experience using engines.

I’ve used engines a fair bit. Here are some of my observations…

As you point out, the underlying tech of a gem (in this case, plus a railtie) seems more suited for library dependencies as opposed to a way of specifying broad application level dependencies.

I can see the initial appeal with engines. They allow you to see dependencies in your code (via gems) without having to write code in a reflective way (e.g using some notion of dependency injection). An additional benefit to this, particularly in large apps, is to leverage this dependency graph to only run a subset of your tests.

I’ve seen two main kinds of engines in the wild. I’ll call them “application level” and “library level”.

The canonical example of a library level engine is devise. You can mount it in any rails app (or engine for that matter). You could even mount it in many different engines.

Worth pointing out that it’s possible to have a library level engine that doesn’t have rails models, views or controllers. In this case, it probably doesn’t need the railtie provided by engines and could just be a gem. The reason I point this out is that once you go down the path of engines, it’s possible folks (understandably) forget or don’t know there are other options!

An example of a well behaved application level engine could be cruddy admin interface. It has its own controllers, views, and maybe even models – but ultimately the dependency goes in one direction. The rails app mounts the admin interface and the admin interface has access to the contents of the rails app. I’m distinguishing it as well-behaved because ideally nothing (or in practice, effectively nothing) in the rails app proper cares about the existence of the admin engine.

An example of a potentially misbehaved application level engine might be the extraction of a subsection of your domain into an engine. For instance, most rails apps are going to have some notion of billing. The code smell I’d look for is an engine with the the absence of controllers or views – or an engine that is never mounted.

If this engine is only ever consumed by the rails app, you probably won’t feel too much pain. But with just the one consumer, what are you really getting from being in an engine? It really isn’t much more than a namespace.

IMO, where things start to go wrong is the moment an application level engine depends on another application level engine. If you’ve opted to pursue this pattern, it’s conceivable that you’ll end up with two consumers of this billing engine – probably with different requirements.

Since it’s a gem with a railtie, you’re stuck giving access to the whole engine to both consumer engines. So what do you do? Well, you probably at least consider splitting the billing engine into two engines.

I’ll concede that you can see these application dependencies as gem dependencies – but is getting that visibility also the source of the problem?

It’s hard to talk in generalities since the sheer size of the app and complexity of the domain is likely to make you feel these pains more or less.

I love the idea of formalizing the module namespace convention. It seems a number of folks have arrived at variations of the same solution. Perhaps a way to make engines better is to canonicalize the module namespace pattern so we stop using engines as a means of organizing our domain – leaving them as a means of organizing our rails application.

And if this convention is formalized, perhaps it makes it easier to build tooling to help solve the more domain specific problems like only running subsets of tests or visualizing domain dependencies.

Note: Purposefully didn’t get into when to extract something into a separate rails app. I do think it’s a relevant part of the conversation when considering how engines can be used. Personally, I would defer defer defer until it’s so blatantly obvious that it’s the right thing to do – if ever.

6 Likes

I’m not sure following what the big players do is the right approach for the typical Rails team.

For smaller teams with less technical depth, a significant challenge would be how to organise code to keep complexity under control. Patterns and conventions are helpful for that.

For big players, a different significant challenge would be how to keep hundreds of developers from clobbering each other on every commit. A strongly separated architecture (eg. engines, microservices) delivers enough value there that it makes sense to dedicate the considerable resources required to design and maintain it.