Kamal environment/credentials issue on deply

rubyonrails-talk
1

I am trying to deploy a fairly basic Rails 8 app with Kamal in a staging environment. Locally, rails s -e staging works fine. In the base deploy.yml I have the default implementation of the rails credentials keys:

env:
  secret:
    - RAILS_MASTER_KEY

In Kamal secrets the corresponding key is set for the staging variant:

RAILS_MASTER_KEY=$(cat config/credentials/staging.key)

And finally in the staging-specific Kamal config (deploy.staging.yml) the RAILS_ENV is set:

env:
  clear:
    RAILS_ENV: staging

When I run a deploy with Kamal (kamal setup -d staging) it fails as follows:

INFO [d31195bd] Running docker exec kamal-proxy kamal-proxy deploy myapp-staging-web-staging --target=“6a1891496841:80” --host=“myapp-staging.example.com” --tls --deploy-timeout=“30s” --drain-timeout=“30s” --buffer-requests --buffer-responses --log-request-header=“Cache-Control” --log-request-header=“Last-Modified” --log-request-header=“User-Agent” on 123.456.789.0

ERROR Failed to boot web on 123.456.789.0

INFO First web container is unhealthy on 123.456.789.0, not booting any other roles

INFO [a4e70f6b] Running docker container ls --all --filter name=^myapp-staging-web-staging-ba2a1f3d60d520d82d53cb7aafb3efbd870c4f2b$ --quiet | xargs docker logs --timestamps 2>&1 on 123.456.789.0

INFO [a4e70f6b] Finished in 0.327 seconds with exit status 0 (successful).

ERROR 2025-12-10T16:27:30.929391116Z bin/rails aborted!

2025-12-10T16:27:30.929459894Z ActiveSupport::MessageEncryptor::InvalidMessage: ActiveSupport::MessageEncryptor::InvalidMessage
2025-12-10T16:27:30.929465590Z /rails/config/environment.rb:5:in ‘’
2025-12-10T16:27:30.929471136Z Tasks: TOP => db:prepare => db:load_config => environment
2025-12-10T16:27:30.929475080Z (See full trace by running task with --trace) 2025-12-10T16:27:34.169510376Z bin/rails aborted!

This error usually means that the incorrect key is being used for Rails credentials, yet, everything I see here shows the the correct environment is being passed (further up the Kamal output is a docker run command that includes --env RAILS_ENV=“staging” so the env variable from the Kamal config is being used) and I know the correct key is being used because I can view .kamal/apps/myapp-staging-staging/env/roles/web.env when I SSH into the remote server and it contains a value for RAILS_MASTER_KEY that matches the value of config/credentials/staging.key locally.

What else might be causing this error in the container on boot that wouldn’t be happening on my local dev machine? What other tests can I run against the docker container to find out what isn’t being configured correctly? Since Rails in the container crashes before starting the server I can’t see what environment it’s printing out (I’m left to rely on the passed --env value I see elsewhere in the docker commands run by Kamal to guess what environment Rails is booting). Is there another way to verify it’s running in staging and not production inside the container?

1 Like
2

Turns out this is a case of how easy it is to forget that Kamal is working with the current version of the repo in git, not the current working directory:

config/credentials/staging.yml.enc was created locally but not yet checked into git! Simply adding the file to git resolves this error.

1 Like