Async callbacks as a first class citizen

rubyonrails-core
1

Summary

We often create callbacks that only trigger a background job. It would be simpler if we could inline the logic for the background job in the model and just mark the callback as async. This would “collapse” the perceived complexity of the implementation, since we won’t need an additional background job class.

I’m not sure how do people feel about this? Would you consider this an anti pattern or a way to reduce boilerplate.

Example

In our codebase we have something along these lines:

class Recording < ApplicationRecord
  after_create_commit :process

  def process
    ProcessImportedRecordingJob.set(wait: 5.seconds).perform_later(self)
  end
end

It would be good if we could do something like this instead and have the same functionality:

class Recording < ApplicationRecord
  after_create_commit :process, async: true

  def process
    # Processing logic
  end
end

Potential Problems

The implementation might be tricky, since we’d need to either:

  1. Find a way to pass a proc or a block to a background job.
    • FWIW I don’t think this is easily solvable.
  2. Dynamically create a background job class and use that to execute the callback
    • This seems like the most viable solution
1 Like
2

While I understand the motivation, I’m a bit skeptical. It’s too abstract about what’s going on under the hood.

The implementation might be tricky

That implies it does too much under the hood. If we do that by ourselves:

class RecordingProcessJob < ApplicationJob
  def perform(recording_id)
    recording = Recording.find(recording_id)
    # Processing logic
  end
end

class RecordingController < ApplicationController
  def create
    recording = Recording.create!(recording_params)
    RecordingProcessJob.perform_later(recording.id)
  end
end

And essentially that’s it. The best thing about this is we know processing will be done in background because we have RecordingProcessJob class.

2 Likes
3

Check out Delayed Extensions of Sidekiq.

It was not specific for callbacks, but the idea is similar.

1 Like
4

I really like this idea. Rails already has dependent: :destroy_async and I’ve long thought this pattern should be extended since this is not the only situation where you want to easily move something from executing in process to executing async.

If your feature was implemented, then I think this dependent destroy syntax should change to: dependent: :destroy, async: true as well.

There is a very similar feature I’ve also wanted where you can easily execute any method of a model in an async manner. For example, maybe your user model has a method “expire_memberships!” and sometimes you want that executed synchronously but other times you want to execute it async. It would be really nice if you could prefix any method with “async” so Current.user.async.expire_memberships!

I don’t mean to hijack your proposal, but I’m mentioning this here because I’ve been meaning to propose it and it’s so closely related to what you’re thinking.

I looked up the implementation of destroy_async and it just uses a single generic job for this. The same implementation could work for your async: true

I happened to bring this up on reddit last week and there was some good discussion of pros/cons and some sample code. This might inspire some ideas for you: Reddit - Dive into anything

5

I’ve been exploring a version of this over in GitHub - kaspth/active_job-performs: ActiveJob::Performs adds the `performs` macro to set up jobs by convention.

In ActiveJob::Performs I’m making a convention of passing a model as the first argument to a job and calling an instance method on it. This is an internal pattern Active Storage and Action Mailbox that I’m hoisting to Rails apps in general.

So with my gem, your example can be rewritten as:

class Recording < ApplicationRecord
  after_create_commit :process_later

  def process
    # …
  end
  performs :process, wait: 5.seconds # Generates `process_later`
end

And this is the equivalent Ruby code that you’d write manually:

class Recording < ApplicationRecord
  class Job < ApplicationJob; end # For any shared configs, target with `performs wait: 5.seconds`
  class ProcessJob < Job
    def perform(recording) = recording.process
  end

  def process
    # …
  end
  
  def process_later
    ProcessJob.set(wait: 5.seconds).perform_later(self)
  end
end

Doing it this way generates a nicer conventional API with process_later that callbacks and other code can call. We also get a space to set up any needed configuration directly, where trying to do that with the async: true API would grow cumbersome.

performs also opens the door to other interesting app-wide patterns.

Say if there’s an Active Record method that you’d like any model to be able to run from a background job, you can set them up in your ApplicationRecord:

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true

  # We're passing specific queues for monitoring, but you may not need or want them.
  performs :touch,   queue_as: "active_record.touch"
  performs :update,  queue_as: "active_record.update"
  performs :destroy, queue_as: "active_record.destroy"
end

Then a model could now run things like:

record = Invoice.first
record.touch_later
record.touch_later :reminded_at, time: 5.minutes.from_now # Pass supported arguments to `touch`

record.update_later reminded_at: 1.year.ago
# Particularly handy to use on a record with many `dependent: :destroy` associations.
# Plus if anything fails, the transaction will rollback and the job fails, so you can retry it later!
record.destroy_later

You may not want this for touch and update, and maybe you’d rather architect your system in such a way that they don’t have so many side-effects, but I find having the option to be interesting!

There’s more examples and options in the README. I’m curious if this strikes a chord with your use-case and if the API makes sense?

1 Like
6

delayed_job (which pre-dates ActiveJob) has some of the ideas here. You could call handle_asynchronously with a symbol that corresponds to an instance method. It would convert that method into an async method (while also still providing access to a sync version).

While convenient, I found the more structured approach to always having a “Job” object was better from a testing point of view. Not saying their wouldn’t be times where a lightweight way to “async” a method would be useful. But most of the time the structured approach of a “Job” object is superior IMHO.

7

@kaspth Thanks Kasper, this is exactly what we were looking for! We’ll actually start adopting this because it solves 90% of what we need. If there’s anything extra, we’ll submit PRs upstream. Hope this lands in Rails one day.

@e_a I think the way to think about this is that it’s about calling a method asynchronously without having to create a job to do so explicitly. The method can be tested with unit tests, with the same ergonomics a job can. Also, being able only to call one method asynchronously is how you’d limit the complexity of the anonymous job.

1 Like
8

@matteeyah that’s perfect, I’m really glad to hear that! And issues or PRs are more than welcome!

A while back there was a discussion about adding class methods support but it stalled a bit, if you hit the need for this feel free to open a new issue and we can talk about it there.

If you’re using Solid Queue you may be interested in this too (I never got around to putting it in the docs) Exception passing hash to `performs` arguments · Issue #8 · kaspth/active_job-performs · GitHub

9

We’ve implemented ActiveJob::Performs in a WIP merge request, and so far we like it! It really helps keep the logic contained in the model - instead of background jobs slowly grow into service objects (which we want to avoid). It’s also as easy to test as testing a background job.

1 Like
10

Yep, you get it! I’m really glad this is clicking for you and your team!

I’d love to see how you’re doing this on an issue over on the ActiveJob::Performs repo, maybe there’s some testing helpers I ought to add?

11

I opened an issue about calculating the job class name Test helper for making sure a `performs` job was scheduled · Issue #10 · kaspth/active_job-performs · GitHub via a method. Seems more future proof than manually typing in the job class name based on the docs.

We’ll run this in production for a couple of days and see how it performs, then report back here with how it went and what the implementation looks like for us.

1 Like
12

Sounds good, I’ll take a look soon!

13

I wrote a version of this back in 2022 with a global wrapper that calls the equivalent of .async under the hood:

The idea is you can do async(Current.user).method, which calls Current.user.to_active_job.method under the hood.

14

The main differences between my gem and Kasper’s (other than his being nicer and having a more robust test suite lol) is that mine only work for ActiveRecord objects and don’t dynamically generate a new job class (they share one abstract job class by default, but you can override that).

That would mess up a good amount of reporting and instrumentation.

The benefit of mine is that instead of just copying and forwarding arguments for the dynamically generated process_later method, I wrote a really basic parser to generate the async_method with the same signature. This just means that during local development you get feedback if you don’t pass the right arguments when calling the async method instead of when the job fails.