Yesterday we upgraded our EC2 servers to a new custom Ubuntu AMI. They are now running ruby 3.1.2 (up from 3.1.1) and connecting to a RDS Postgres 14.2 database (up from 13.2). We remained on the same Rails version (we follow main, currently on ref 55cee0).
A couple of hours later, we noticed that every link we had sent on emails and sms messages that included a model’s signed_id were broken. ActionText, which internally relies on MessageVerifier was also broken.
Checking one of the affected models, we noticed that the signed_id had changed. Same happened with ActionText global_signed_id. We double checked our credentials, and there were no changes in the secret key base.
I’ve read through the signing code, but I’m not familiar with security code, so I don’t know how what might have caused the change.
TL:DR: Is there a change on server software, gem version, ruby version, or anything else that can cause signed_id to change when the signed_key_base did not?
@brenogazzola I’m also very interested if you found anything. Our ActionText embedded images all broke recently and I didn’t get to the bottom of why.
NB: the default value of Rails.application.config.active_support.key_generator_hash_digest_class and Rails.application.config.active_support.hash_digest_class changed in Rails 7, maybe you kept the same Rails version but deleted your new_framework_defaults_7.0.rb or otherwise modified those values? (1, 2, 3, 4)
Below is the little info/questions I gathered about rotating secret_key_base and any other elements that might affect signed IDs:
The same thing happened to me. I am using signed_id to send many URLs by email and also for authentication. All sessions were deleted, although that is the least of the problems.
Did you find out the cause of the problem?
I tried adding a signed ID non-regression check as suggested:
But the mere addition of this check resulted in the signed ID getting changed. Any idea what could be going on? @brenogazzola could you say a bit more on how you wrote your check? At least it’s progress, in the sense that the described issue got reproduced: signed_id changes even though the secret_key_base does not.
Below is the initializer code I tried:
# config/initializers/my_initializer.rb
Rails.application.reloader.to_prepare do
# Ensure signed IDs remain stable (non-regression check)
if Rails.env.production?
record = MyModel.select(:id).find_by_slug('my slug') # load a well-known record
# hard-code the expected signed IDs (values obtained by running
# `MyModel.select(:id).find_by_slug('my slug').signed_id` in the Rails console)
signed_ids = {
"staging id" => "ey...=--...fa02", # for staging
"prod id" => "ey...=--...8330" # for production
}
expected = signed_ids[record.id]
actual = record.signed_id
if actual != expected
puts "Signed ID changed! (record ID = #{record.id})"
puts "- Expected: #{expected}"
puts "- Actual: #{actual}"
Sentry.set_extras(expected_signed_id: expected, actual_signed_id: actual)
Sentry.capture_message("Signed ID changed!")
else
puts "Signed ID did not change (record ID = #{record.id})."
end
end
end
The only part I might find suspicious is loading a DB record during app initialization. Is it not OK? How then to run the check otherwise?
This is my initializer code, placed in config/initializers/signed_id_check.rb
Rails.application.config.after_initialize do
if Rails.env.production? && User.find_signed("YOUR SIGNED ID HERE").nil?
raise ArgumentError, "Something has changed the signed ids. DO NOT DEPLOY"
end
end
This halts my deploys in case of a failure because, before the newer code is downloaded to the servers and puma/sidekiq restarted, a “control” server runs a rails zeitwerk:check to ensure everything is fine and trips this.
Just to keep track of things that don’t break signed IDs: we’ve upgraded from Ubuntu 20.04.4 to 22.04.1, while staying in the same version of rbenv ruby and postgres. Everything is fine.
We’ve got incoming everyone. Was checking the new_framework_defaults_71 file and saw they are changing message encryptor’s and message verifier’s serializers.