ActiveJob - N+1 when serialising/deserialising an array of ActiveRecord objects

[DanielGilchrist]

I’ve ran into a problem a few times in our Rails codebase where a developer will pass an array of ActiveRecord objects as an argument which will serialise into an array of Global IDs, which are then queried one by one when deserialising inside the job resulting in an N+1. This gist replicates this scenario: ActiveJob serialise/deserialise with an array N+1 replication · GitHub

This occurs because of how serialisation and deserialisation currently works with arrays in ActiveJob. When serialising a value that is an Array, ActiveJob simply maps over the array and calls serialize_argument which will go through and convert the records to a Global ID hash:

activejob/lib/active_job/arguments.rb

Upon deserialisation, ActiveJob will do basically the same thing except call deserialize_argument for each record:

activejob/lib/active_job/arguments.rb

This results in each Global ID being deserialised with GlobalID::Locator.locate which results in an N+1

activejob/lib/active_job/arguments.rb

This isn’t really a bug, just an unfortunate side-effect of how serialisation/deserialisation currently works for arrays. An obvious workaround for this would be to simply pass an Array of IDs and query for them manually in the job, but it would be great if ActiveJob could be “smarter” here.

Two potential solutions I’ve thought of are:

1. We leave serialisation as-is, and update derserialize_argument when checking for Array to see if the records in the array are global ID hashes, if they are we extract the IDs and use GlobalID::Locator.locate_many

The main problem I see with this is we have to worry about cases where an array of mixed objects are passed. This means that we can’t simply check the first value in the array to then decide whether we handle the case of an array of Global IDs, because there could be at least 1 other object in the array that isn’t a Global ID. If we check the whole array then we risk unnecessary performance overhead.

2. We add a custom serialiser for ActiveRecord::Relation objects into ActiveJob

The way this would work is we would add a new key GLOBAL_ID_MANY_KEY similar to GLOBAL_ID_KEY. Then on serialisation we would execute the query, build a Global ID hash for each, and then put it in a hash with that key as a value. When deserialising we would check for a hash with that key, extract the Global IDs and then query for all of the records at once with GlobalID::Locator.locate_many.

The main problem I see with this is the behaviour may be somewhat surprising, especially considering the argument would essentially be converted into an array of records when accessed inside of the job.

Has anyone else run into this problem and have any other creative solutions? Keen to hear any thoughts on the topic

[joshuay03 (Joshua Young)]

I quite like the idea of 2. I personally don’t think it would be too surprising, given that currently, relation serialisation is not supported by default i.e., will raise:

ActiveJob::SerializationError: Unsupported argument type: ActiveRecord::Relation

It’s technically possible to implement by serialising the relation values e.g. where_values and rebuilding it on the other end, but I really don’t see that happening given how complex relation building can get, and all the edge cases that could arise from this. It would also require quite a bit more coupling between AR and AJ. So, I think your proposed approach is much more ideal.

You could also support :includes with .locate_many: GitHub - rails/globalid: Identify app models with a URI. Essentially, execute the query without includes values when serialising, embed the values in the hash, and use them during deserialisation.

I’m keen to hear what Core/Committers think about this.

[seanpdoyle (Sean Doyle)]

@joshuay03

You could also support :includes with .locate_many: GitHub - rails/globalid: Identify app models with a URI. Essentially, execute the query without includes values when serialising, embed the values in the hash, and use them during deserialisation.

I’ve opened Introduce `ActiveJob::Base.locator_options` by seanpdoyle · Pull Request #50341 · rails/rails · GitHub to explore building-in first-party integration with GlobalID’s :includes option. There’s some overlap in the kind of problem its trying to solve with regard to N+1 queries (through Strict Loading violations, in particular).

[e_a]

@joshuay03

It’s technically possible to implement by serialising the relation values e.g. where_values and rebuilding it on the other end, but I really don’t see that happening given how complex relation building can get, and all the edge cases that could arise from this.

I did that exactly on a previous project and this previous discussion has my implementation. You are correct that there are edge cases that are not worth handling but for the common case it was sufficient and if someone tried to pass a relation that has one of those edge cases an exception was raised. Not a 100% solution but something that was good enough for my needs and might be for many needs.

[joshuay03 (Joshua Young)]

@DanielGilchrist Core has recommended sending a PR for the relation solution so we can kick off a discussion. There’s potential here to both implement that and support locate_many in @seanpdoyle’s APIs, instead of trying to serialise any other values (i.e., includes) from the relation itself.