The application described in this long-winded post does not live on the Rails “happy path”. The problems it encounters are unlikely to affect most Rails applications. The frustrations described herein, however, are most definitely real. I also try to discuss a bit beyond the issues encountered, into the solutions we’ve attempted or implemented. Some of these — perhaps many — are outright hacks, prone to fragility, or otherwise poor code. While I would be happy to hear about documented approaches or APIs that I’ve missed, I would ask readers to hold their judgements — the application here is (as all software is) a series of considered tradeoffs and balanced concerns.
Before I get into this, I want to acknowledge that Rails is great, and the core team is inspiring — the work you all have done and are doing is what allows so many of us to be successful. So, thank you all for everything you’ve done, and for taking the time to engage the community around our pain.
The Application
I work for an organization whose primary application is a CMS for authoring K-12 textbooks. This application is used exclusively by organization employees and their partners, meaning that many typical web application concerns — high concurrency, malicious input sanitization, etc. — aren’t high priority needs for us.
All of our development happens in Docker (in an Alpine Linux environment, usually running on macOS), running Rails 6.0.3 on Ruby 2.6.4. We’re backed by a PostgreSQL database, with Redis and Sidekiq for background jobs. Our production and staging instances are managed by Heroku.
Our application manages data across multiple curricula in a single database, structuring the data in a freeform tree (backed by PostgreSQL’s ltree plugin, for those curious). Our curricula don’t uniformly follow the same structural convention (e.g. Module → Unit → Lesson), so our trees may be unpredictably deep in places and unexpectedly shallow in others.
Despite sharing the same base models (and database tables), we’ve architected our curriculum code (which we refer to as “projects”) in such a way that they are isolated from one another, and use STI to achieve specialization. This lets us, for example, encode the differences between Lessons in different curricula, and between Lessons and Units in the same project. In practice, our projects are something like little Rails Engines — directories of views, models, assets, and configuration that follow a regular structure which our application understands.
Because the review and editing patterns are so consistent between models, we handle all of our CRUD operations on those records through a single controller, which delegates to a model-specific form object.
At the tail end of the process, we generate the various textbooks we produce, rendering massive HTML pages that traverse most (if not all) of the given tree (along with any associated embedded content and tags). This HTML then gets passed off to various processes that refine it into its terminal forms.
Spring
We ended up removing Spring fairly early in the application’s life. Ultimately, we found that we were restarting the Docker container frequently enough in development that the advantage of having a background process that preloading the application was being lost to the initial boot overhead.
I also have personally lost days of development time to Spring cache issues, where I’m trying to diagnose why the change I’ve made to the code hasn’t produced the logical results I’m trying to effect — and I’m not the only one on my team. This, in turn, leads to a distrust of Spring and a preference towards the more reliable behavior of restarting the Docker container to avoid the issue. Removing Spring made our effective development work faster by avoiding the startup overhead and reducing the number of times we felt the need to restart the container, even as our individual rails command runs lost speed. YMMV.
I feel like Spring exists in a somewhat unfortunate position — it’s a background process that’s a part of your development environment, with the specific aim of being an invisible performance boost that developers don’t have to think about. At the same time, owing to cases where Spring is too good at caching, Spring becomes something that everyone has to constantly think about in case they need to work around it. This has been something I’ve witnessed on every Rails project I’ve been a part of, and why I see Spring consistently being disabled on those projects.
Sprockets
We’ve also stripped Sprockets out of our application. Having worked with Webpack extensively on other projects, I’ve become familiar with the nuance of Webpack configuration (and enamored of hot reloading both Javascript and CSS). While Sprockets offers a simpler experience for CSS composition, Webpack offers a better experience for development (quirks notwithstanding).
There are a few obstacles to setting this up correctly — our application’s directory structure being only one of them — but it’s entirely possible set up Webpacker to resolve CSS and image files nearly identically to how it handles Javascript.
Running Rails without Sprockets, however, has its own challenges — especially if you’re using Engines. At present, there is no good story for Engine-based assets that doesn’t involve either static compilation or Sprockets, and most Engines have an implicit dependency on Sprockets. While Rails will let you choose to opt out of Sprockets at application creation time, there’s no indication that this decision is also opting you out of some subset of useful tools (e.g. PgHero).
Webpacker
Webpack (and Webpacker) are not without their own WTFs, but we’ve been content to set up our own conventions and work within them. (So much so, in fact, that when we needed to do a visual facelift on a secondary application we manage, our front-end engineer specifically requested that we reproduce the Webpacker setup on that application!) The biggest “complaint” we’ve had about Webpacker had to do with how its helpers function. If you set the extract_css option to false, any CSS required by your Javascript will be injected from Javascript; if it’s true, a separate CSS file will be generated that needs to be loaded. If you’re using different values in production and development (e.g. hot reloading CSS in development), this means that you need to call both javascript_pack_tag and stylesheet_pack_tag for the same entry points in order to get somewhat consistent behavior.
To work around this in our application, we wrote our own helper that builds inclusion tags for all JS and CSS files generated for a named entry point. (Passing options to each is a little clunky, but it’s also not something we do often.)
def asset_pack_tags(*names, js_options: {}, css_options: {})
manifest = current_webpacker_instance.manifest
js_entries = names.flat_map { |name| manifest.lookup_pack_with_chunks(name, type: :javascript) || [] }.uniq
css_entries = names.flat_map { |name| manifest.lookup_pack_with_chunks(name, type: :stylesheet) || [] }.uniq
js_tags = javascript_include_tag(*js_entries, **js_options)
css_tags = stylesheet_link_tag(*css_entries, **css_options)
return js_tags + css_tags
end
Zeitwerk
There is a lot of inconsistent documentation around autoloading in Rails. Formally (I believe), the “official” interface for modifying autoload paths is Rails.application.config.paths.add — but there is still plenty of advice promoting modifications to autoload_paths and eager_load_paths. As a bonus, IIRC, config.paths.add is documented as taking an autoload option, but it’s not passed by any of the Rails code, and paths are autoloaded without it.
The recent addition to Zeitwerk of the collapse method was a fantastic boon to us, allowing us to simplify paths to our project models from projects/<project_name>/models/<project_name>/project_model.rb — so glad to see that change!
There are still a couple rough edges, though. Our projects, like most gems, have files that don’t follow the same filename → constant convention. In gems, these are usually the lib/<library>/version.rb files; in our projects, they live at projects/<project_name>/config/project.rb. While Zeitwerk can support these files through the use of a custom inflector, the one-off nature of these files makes that feel fairly overkill. It seems like it would be generally preferable to be able to declare explicit mappings for these exceptions. That would provide useful flexibility, allowing minor infrequent exceptions to be quickly defined with a method call, and allowing alternative conventions to be implemented via a custom inflector.
ActiveStorage
Like many applications we store user-uploaded assets (like images) in S3 in our non-development environments. In development, we’re perfectly content to use our local disk for storage. Configuring ActiveStorage to do this is simple and straightforward, as it should be.
Given the complexity of data we work with, it’s easiest for us to simply load a backup of the production data server into our development databases; this is where we get into problems with ActiveStorage. Because all of the bookkeeping for AS is done in the database, our development environment knows the keys of all of the production assets … but will never find them on disk (unless we also copied everything from S3 to local disk as well).
For our use case, what we actually wanted was a way to specify fallbacks for failed lookups. We would never want development to update the production S3 bucket, for example, but if it could read from it, that would be helpful. On the other hand, we would definitely still want reads and writes to continue to work in development, so we couldn’t simply point to the production database with a read-only account. ActiveStorage provides a MirrorService, but nothing like we were looking for…
So we built one.
# Usage:
#
# local:
# service: Disk
# root: <%= Rails.root.join("storage") %>
#
# remote:
# service: S3
# region: us-east-2
# bucket: <%= ENV['S3_BUCKET'] -%>
# access_key_id: <%= ENV['S3_ACCESS_KEY'] ) %>
# secret_access_key: <%= ENV['S3_SECRET_KEY'] ) %>
#
# local_with_fallback:
# service: ReadReplica
# primary: local
# replicas: [ 'remote' ]
class ActiveStorage::Service::ReadReplicaService < ActiveStorage::Service
attr_reader :primary, :services
delegate :upload, :update_metadata, to: :primary
delegate :delete, :delete_prefixed, to: :primary
delegate :path_for, :url_for_direct_upload, to: :primary
def self.build(primary:, replicas:, configurator:, **options)
primary = configurator.build(primary)
replicas = replicas.map { |name| configurator.build(name) }
return self.new(primary: primary, replicas: replicas)
end
def initialize(primary:, replicas:)
@primary = primary
@services = [ primary, *replicas ]
end
def download(key, &block)
service_for(key).download(key, &block)
end
def download_chunk(key, range)
service_for(key).download_chunk(key, range)
end
def exist?(key)
@services.any? { |service| service.exist?(key) }
end
def url(key, **opts)
service_for(key).url(key, **opts)
end
private
def service_for(key)
@services.find(-> { @primary }) { |service| service.exist?(key) }
end
end
Credit where credit’s due: while this wasn’t baked into Rails, and while the documentation is a bit sparse, adding a custom ActiveStorage Service was generally straightforward.
The other issue we’ve had with ActiveStorage is a relatively minor nuisance, but one that is worth mentioning all the same. Our image attachments are a mix of hi-res raster (usually PNG) and vector (usually SVG) images. Naturally, sending dozens of multi-megabyte images over the wire during the editing process is not particularly useful, which is where ActiveStorage Variants come in. However, we cannot simply call Attachment#variant and get reasonable results — that method raises an exception if you attempt to call it for an Attachment that is not variable? (like an SVG). Consequently, our templates end up littered with fragments like this:
<%= image_tag polymorphic_path(record.image.variable? ? record.image.variant(...) : record.image) -%>
This is doubly problematic, since forgetting to check variable? before calling variant will look correct often enough that changes can get at times pass through QA without having triggered the failure case. In applications where the attachments are truly unpredictable user input, I would expect the issues to be even more common.
What would feel useful here is a method that allows you to express the intent to (e.g.) use a thumbnail-sized variant, and left the determination of how to proceed with the library. Such a method would be prone to a different type of error (namely, handling cases where a constrained image size was specified, but the unconstrained image is served and breaks the layout) but that concern is presentational and non-fatal, which seems preferable for an error case this well-hidden.
ActiveRecord :: Associations without Foreign Keys
This is a minor use case presenting major difficulties. As mentioned, we’re using PostgreSQL’s ltree extension to model our content hierarchy — this employs a “materialized path” to denote a node’s location within a tree (e.g. Root.Photos.Science.Astronomy). The extension allows the database to sensibly index the relationship between nodes in the same tree, so queries for descendants are just as fast as a query for a parent. What this means, however, is that we don’t have a foreign key for our relationships, just an SQL expression.
ActiveRecord’s association macros are great, but they have a baked in assumption (and usually rightly so) that there are two columns that can be compared between tables to perform a join. The workaround isn’t pretty, but it’s functional.
fk = self.name.foreign_key
has_many :children, -> (node) { unscope(where: fk).where(project: node.project).where("subpath(path, 0, -1) = ?", node.path) }
This approach works, but it introduces a couple of additional problems.
ActiveRecord :: Creating off Associations without Foreign Keys
Because our newly-minted association doesn’t have a foreign key, Rails can’t (and shouldn’t be expected to) figure out how to ensure that it’s setting up parentage correctly. Instead, that’s something that quite reasonably falls to our application. If we have a has_many relationship, it’s relatively easy to patch that in:
has_many :children, -> {...}, before_add: assign_parent
private def assign_parent(parent)
self.path = [ parent.path, self.name ].join('.')
end
If the relationship is a has_one, we have a different problem — has_one doesn’t support before_add, or any comparable callback. Maybe that’s fine (he said, hopefully), has_one just creates specialized instance methods on the record (e.g. build_<association>) method, so those can be overridden.
Remember how I said we’re using form objects to handle all of our content edits? Those form objects also edit and create nested records, and to do that, they rely on Association#build and Reflection#build_association. Even if we did just override build_child, our form objects would have the same problem. Unfortunately, fixing this is ugly.
class TreeNode
has_many :child, -> {...}
reflect_on_association(:child).define_singleton_method(:association_class) do
MyHasOneChildAssociation
end
class MyHasOneChildAssociation < ActiveRecord::Associations::HasOneAssociation
def initialize_attributes(child, except_from_scope_attributes = nil)
super
child.set_parent(@owner)
end
end
end
At this point, both children.build and build_child work properly, but that’s not quite the end of the story. children.create and create_child (along with the ! counterparts) fail after trying to assign an attribute to tree_node_id! The TL;DR is that those methods end up calling Association#set_owner_attributes, which tries (and fails) to assign the foreign key and type (when appropriate). We’ve worked around this by creating custom subclasses of both HasOneAssociation and HasManyAssociation that do more appropriate things in set_owner_attributes, but it fundamentally feels as though there’s a missing interface here. Potentially:
- An option on the macro for customizing the
association_classfor the association. - An option on the macro for specifying a class or object that overrides the implicit condition for finds and the implicit behaviors for creates.
- Meta-association methods for creating custom associations that behave like the built-in associations (e.g. build/create, reflection, preloading, etc.).
ActiveRecord :: PostgreSQL Generated Columns
The previously described associations would be easier to work with in Rails if we just had a foreign key on our table. Our path column is prone to change, however, adding a second column with derived data seems both redundant and error-prone. We could use a view (materialized or otherwise), but then we don’t have the ability to write changes back. Database triggers or functions are technically also an option. The “best” integrated option, however, would be a virtual column — Rails added support for defining them in migrations for MySQL and MariaDB back in Rails 5.1.0.
PostgreSQL also recently built out a similar feature in PostgreSQL 12, which they call “generated columns”. Sadly, I have seen no indication that Rails will be adding that functionality for the PostgreSQL adapter any time soon.
ActiveRecord :: Preloading
Because (as you may have noted above) we specified our children association with a scope that took a node argument, Rails won’t allow us to use the built-in preloading methods. While that’s disappointing, it’s also completely understandable — sorting that out isn’t something Rails can do on its own. It is, however, something that can be sorted out by someone familiar with the data domain.
Our needs are actually larger than that: we don’t know up front how many levels of children we should be preloading, but we do know how to trivially query the database for the entire tree. To that end, we’ve actually built out functionality that will “preload” associations for the entire tree. It looks something like this:
class TreeNode
def self.preload_descendants
extending(PreloadDescendants)
end
module PreloadDescendants
def load
super
table = model.table_name
loaded_paths = @records.pluck(:path)
return if loaded_paths.empty?
# Ensure that we maintain the project condition.
descendants = model.where(where_values_hash.slice('project'))
# Load all descendants of the queried nodes.
descendants = descendants.where("#{table}.path <@ ARRAY[?]::ltree[]", loaded_paths)
.where.not(path: loaded_paths)
# Load the tree from the bottom up, preserving the other order clauses.
descendants.reorder!(Arel.sql("NLEVEL(#{table}.path) DESC"), *order_values)
# Ensure that we do the same eager loading here that we did for our
# original results.
descendants.includes!(*includes_values) if includes_values.present?
descendants.preload!(*preload_values) if preload_values.present?
descendants.eager_load!(*eager_load_values) if eager_load_values.present?
# Ensure that we reuse the same extension modules that we had earlier.
# Obviously, we should omit ourselves, since our work is already done.
descendants.extending!(*(extending_values - [PreloadDescendants]))
nodes = descendants + @records
nodes_by_parent = Hash.new { |h,k| h[k] = [] }
# Because we've explicitly ordered our result set on `NLEVEL() DESC`, we will visit
# all of the most deeply nested nodes first. In particular, we will be guaranteed
# to visit all children of a node before we visit their parent. Rails will also
# help guarantee that by freezing the array we send to `load_records` so that any
# subsequent writes will cause errors.
nodes.each do |node|
nodes_by_parent[node.parent_path] << node
association = node.association(:children)
association.target = nodes_by_parent[node.tree_path]
association.loaded!
end
return self
end
end
end
Worth noting:
- We’re using
extendingto add a module to the Relation.- That module overrides
load, giving us a convenient “hook” for when the query is actually executed.
- That module overrides
- We construct a nearly identical query to the one implied by the Relation.
- We manually “preload” the association by setting
targetand callingloaded!.
This, conveniently, works! Mostly. It definitely feels dirty, though.
We came to this solution when we found that we simply didn’t have a better option. Given a bucket of nodes, we know how to associate them, but there didn’t seem to be a better way to “hook” the Relation between when it loaded the data and when that data was consumed. We certainly could pass the relation to a service object that either preloaded each relation as above or (more cleanly) provided an interface for requesting relatives of a given node — and we did try it — but that interface is both less convenient and less conventional.
An interface that would have served us well here would be a callback for when a relation has loaded (ideally once per table queried!), and an “official” mechanism for populating the dataset for an association and/or relation.
ActiveRecord :: Extended Scopes on Preloaded Associations
This is a minor issue that I ran up against while I was working on a preloading refactor.
Consider the following code:
class Role; end
class User
has_many :roles
def self.preload_roles
includes(:roles)
end
end
class Thing
belongs_to :user
end
# This loads the Thing, the Thing's User, and that User's Roles.
thing = Thing.includes(:user).merge(User.preload_roles)
# This is does the same thing.
Thing.belongs_to :user_with_roles, -> { preload_roles }
thing = Thing.includes(:user_with_roles).first
So far, so good. We can use merge to concatenate scopes, and the concatenated scopes appear to be associated with the correct objects. Now consider:
User.has_many :fancy_roles, -> { where(type: 'fancy') }
def User.preload_roles
includes(:roles).extending(PopulateFancyRoles)
end
module PopulateFancyRoles
def load
super
# Assume this is going to do something relevant, like populating
# `fancy_roles` from the preloaded `roles` association.
warn self.model
end
end
# What do you expect these to log?
Thing.includes(:user_with_roles).first
Thing.preload(:user_with_roles).first
Thing.eager_load(:user_with_roles).first
Thing.includes(:user).merge(User.preload_roles).first
Thing.preload(:user).merge(User.preload_roles).first
Thing.eager_load(:user).merge(User.preload_roles).first
