I would like to use Rails with AWS Secrets Manager (ASM) or Hashicorp Vault, where our ActiveRecord (Postgres) password is stored in ASM or Vault, and they rotate the password every day. I’d like Rails to look in ASM or Vault, or in a local file, or pretty much any supported place would be fine, for the latest credentials before making a new Active Record connection.
An example of what I mean is - suppose the ActiveRecord password is “foo” in ASM when the application starts. Then ASM rotates it to “bar”. Rails lazily is spinning up more connections to ActiveRecord whenever it needs them, but continues using the “foo” password, so new connections fail.
Ideally, it would be possible for Rails to somehow learn that “bar” is now the current password.
If this doesn’t exist, it would be a great feature request! If it does exist, I would love a tip or link on how to implement it.
This won’t totally answer your question, but there are two different things you have to consider:
Where does Active Record get the password? Common way is from the UNIX environment and if you set DATABASE_URL, Rails picks it up. If you want Rails to grab it some other way e.g. via an API call, that will be more complicated and may not be directly supported out of the box
Allowing both old and new passwords to work simultaneously. This is critical because during the rotation, you will have some requests/processes running using the old password and some using the new. It’s going to be pretty hard to do a clean cutover from old to new without both being acceptable for a short duration.
The specifics of how you solve these depends on your hosting and infra setup.
Yeah, I think there could be multiple ways it could grab an updated value for the password. It could be a direct call from the application to an external API, or to a local file. I like the idea of it being delivered as an environment variable, but am I right that just generally speaking, it’s not possible to change the environment variables for a process that’s already running? proc - change environment of a running process - Unix & Linux Stack Exchange has a discussion of it here.
Or, can you give a more specific example of how that might work without having to restart the application? That’s what I’m trying to avoid.
Totally great point, yes, allowing them both to run for a period would avoid that gap that we would statistically hit between rotation.
I think I missed the requirement to not restart the app. In my experience, the app restarts to get updated environment variables. At a high level it would go something like this, assuming you had a load balancer and two Rails processes (two dynos in Heroku parlance)
Rails 1 and Rails 2 are running with Postgres (PG) password oldpw
PG is configured to also allow access via password newpw
Environment configuration is changed, and DATABASE_URL is now using newpw (will explain how I have done this specifically below)
Rails 3 is started up with new environment config. Connects to PG using newpw
Rails 1 is spun down, finishing its requests and disconnects from PG
Rails 4 spins up using new config and connects to PG using newpw
Rails 2 spins down, disconnects from PG
PG’s oldpw password is removed
The process cycling maintains your needed capacity (at least 2 Rails processes) while the switch happens. If you are using some sort of container-based thing, this is how I have seen it done and I think is common. So yeah, you essentially restart the servers with the new configuration.
I am not sure how to do this without restarting Rails, but since there is a connection pool and the connections can be recyled, it should be possible. How easy or how supported…I’m not sure, but this bit of Active Record is fairly approachable codewise, so it might not be too bad to find where it stores the connection info and update it then force a reconnect.
I’ve probably already way over-answered not-the-exact question you have, but for completeness, here is how I have seen the environment managed.
We had a custom app that orchestrated Docker containers. When you launch a container, you tell AWS’ ECS in our case what the UNIX environment was. We stored the “correct” values encrypted in a database, so when a new container was spun up, we’d decrypt the environment, hand it to the container and it’s off to the races. Rotating a key like this was merely updating the encrypted value in the database and either forcing a recycle of containers or waiting for it to naturally occur (we tended to restart everything at least once a day, just like Heroku)
Thanks @davetron5000 for your detailed response! I like the approach you’re suggesting. If we can pull it off the right way, it could mean that there’s no application-side work to make new passwords consumable. I’ll give that some thought given our particular architecture.
