Rails `signed_id` changes even though the signed_key_base does not

Yesterday we upgraded our EC2 servers to a new custom Ubuntu AMI. They are now running ruby 3.1.2 (up from 3.1.1) and connecting to a RDS Postgres 14.2 database (up from 13.2). We remained on the same Rails version (we follow main, currently on ref 55cee0).

A couple of hours later, we noticed that every link we had sent on emails and sms messages that included a model’s signed_id were broken. ActionText, which internally relies on MessageVerifier was also broken.

Checking one of the affected models, we noticed that the signed_id had changed. Same happened with ActionText global_signed_id. We double checked our credentials, and there were no changes in the secret key base.

I’ve read through the signing code, but I’m not familiar with security code, so I don’t know how what might have caused the change.

TL:DR: Is there a change on server software, gem version, ruby version, or anything else that can cause signed_id to change when the signed_key_base did not?

1 Like

@brenogazzola What did this end up being? Was it something with Rails.application.key_generator?

@brenogazzola I’m also very interested if you found anything. Our ActionText embedded images all broke recently and I didn’t get to the bottom of why.

NB: the default value of Rails.application.config.active_support.key_generator_hash_digest_class and Rails.application.config.active_support.hash_digest_class changed in Rails 7, maybe you kept the same Rails version but deleted your new_framework_defaults_7.0.rb or otherwise modified those values? (1, 2, 3, 4)

Below is the little info/questions I gathered about rotating secret_key_base and any other elements that might affect signed IDs:

The same thing happened to me. I am using signed_id to send many URLs by email and also for authentication. All sessions were deleted, although that is the least of the problems. Did you find out the cause of the problem?

@sdubois Unfortunately no. I ran out of time to debug the problem, so I just added an initializer to block deploy if changes again.

My current hypothesis is that a change in openssl versions was the cause. We went from 1.1.1f-1ubuntu2.12 to 1.1.1f-1ubuntu2.13

I think it is not that. I noticed the problem on my local machine immediately after running bundle update. Unless I’m missing something.

Which gems did you see get updated? Which versions?

I tried adding a signed ID non-regression check as suggested:

But the mere addition of this check resulted in the signed ID getting changed. Any idea what could be going on? @brenogazzola could you say a bit more on how you wrote your check? At least it’s progress, in the sense that the described issue got reproduced: signed_id changes even though the secret_key_base does not.

Below is the initializer code I tried:

# config/initializers/my_initializer.rb
Rails.application.reloader.to_prepare do
  # Ensure signed IDs remain stable (non-regression check)
  if Rails.env.production?
    record = MyModel.select(:id).find_by_slug('my slug') # load a well-known record
    # hard-code the expected signed IDs (values obtained by running 
    # `MyModel.select(:id).find_by_slug('my slug').signed_id` in the Rails console)
    signed_ids = {
      "staging id" => "ey...=--...fa02", # for staging
      "prod id" => "ey...=--...8330" # for production
    }
    expected = signed_ids[record.id]
    actual = record.signed_id
    if actual != expected
      puts "Signed ID changed! (record ID = #{record.id})"
      puts "- Expected: #{expected}"
      puts "- Actual:   #{actual}"
      Sentry.set_extras(expected_signed_id: expected, actual_signed_id: actual)
      Sentry.capture_message("Signed ID changed!")
    else
      puts "Signed ID did not change (record ID = #{record.id})."
    end
  end
end

The only part I might find suspicious is loading a DB record during app initialization. Is it not OK? How then to run the check otherwise?

This is my initializer code, placed in config/initializers/signed_id_check.rb

Rails.application.config.after_initialize do
  if Rails.env.production? && User.find_signed("YOUR SIGNED ID HERE").nil?
    raise ArgumentError, "Something has changed the signed ids. DO NOT DEPLOY"
  end
end

This halts my deploys in case of a failure because, before the newer code is downloaded to the servers and puma/sidekiq restarted, a “control” server runs a rails zeitwerk:check to ensure everything is fine and trips this.

Thanks @brenogazzola! That does the trick.

Just to keep track of things that don’t break signed IDs: we’ve upgraded from Ubuntu 20.04.4 to 22.04.1, while staying in the same version of rbenv ruby and postgres. Everything is fine.

It seems that this is related to the turbo-rails update to version 1.1.1.

There is this issue:

Security: signed_stream_verifier_key initializer forces KeyGenerator to SHA1 rather than respecting Rail’s actual defaults · Issue #325 · hotwired/turbo-rails (github.com)

This pr fixes it:

Rails default key generator hash digest class is respected by aaronjensen · Pull Request #335 · hotwired/turbo-rails (github.com)

The default Rails 7 key generator in SHA256 is respected. Previously turbo-rails forced it to SHA1.

Apparently this caused the signed_id to change when updating the turbo-rails gem.

I’m afraid they will revert the change. I don’t think it’s good to roll back to SHA1 either.

This also produced this issue:

After updated to 1.1.1, ActionText attachments are not showing · Issue #340 · hotwired/turbo-rails (github.com)

Can anyone confirm this as well? Thanks

1 Like