Scaling ActiveStorage "Immediate" Variants

rubyonrails-core
,
1

ActiveStorage currently allows variants to be created on demand (by default) or in the background (using preprocessed). This can be problematic for high-traffic applications. For instance, immediately after an attachment is created, thousands of requests could be made for a single variant, causing both the controller and background jobs to race in creating the same variant.

To give more control over variant creation, we propose an immediate option to allow variants to be created simultaneously with the attachment:

  has_one_attached :avatar do |attachable|
    attachable.variant :thumb, resize_to_limit: [4, 4], immediate: true
  end

In this case, when the avatar is created, the same process will immediately create a variant ensuring that it is available without delay and only once process is responsible for the creation.

I’ve created a PR here. I’ve avoided changing or refactoring existing patterns but am open to discussion!

This will be the first of several PRs to assist others managing high-volume traffic of public assets.

2

If you haven’t already you might want to join the official rails discord which I think is used mostly for asking for PR reviews and is more actively monitored by the core team than GitHub PRs. (Take with a grain of salt). But also just wanted to throw my two cents out there…

I personally think a sane default would be utilizing a cache based concurrency control mechanism to check if the job exists. For example, adding the following to TransformJob

before_enqueue do |job|
  if Rails.cache && Rails.application.config.example_config_flag
    cache_key = [job.class.name, job.arguments.join("/")].join("/")
    fail if Rails.cache.exist? cache_key
    Rails.cache.write cache_key, true
  end
end

after_perform do |job|
  if Rails.cache
    Rails.cache.delete [job.class.name, job.arguments.join("/")].join("/")
  end
end

I think this is a more sane default because blocking writes to process those variants is probably a worse alternative for the majority of apps that don’t want the described job stampeding. But the default of job stampeding is also less than ideal, IMO.

3

Hello Tom, I think I ran into a similar case last year and I decided to handle it by exposing the hidden state of “currently being processed”:

I am very much in favor of improving the UX in this area, as this problem definitely isn’t going away.