Suggestion for a new Rails environment that can be used exclusively for automation testing of functional/system tests

With the latest merge into the master branch of this https://github.com/rails/rails/pull/26703, it seems that the core framework is heading in a direction where functional/system tests are also being accepted in a more formal way. If this is the case, imo, it would make sense to also have a RAILS_ENV that is separate from the ubiquitous ‘test’ env. In all my RoR projects where we have written automated functional tests using some framework, we have had to duplicate ‘test’ into something else. Would the community think that such a formalization makes sense?

tx,
Vijay

See also this thread:

https://groups.google.com/d/topic/rubyonrails-core/kqKoJHcQu9U/discussion

…wherein is discussed (without any official response from core contributors) many of the warts and limitations of RAILS_ENV in a modern, cloud-native-y, 12-factor-y world.

– Chad

I highly recommend against “custom” environments.

I disagree with using environments for configuration (such as database connections), they should be used for behavior.

Here’s a thing I wrote about the problems you can get into by solving problems by creating new “env” types: https://devcenter.heroku.com/articles/deploying-to-a-custom-rails-environment

While i’m talking about “staging” as an env, I think it’s generally applicable to any non development/production/test environments.

I also don’t think the problem you’re trying to solve with a custom environment is fully formed. I’ve not found a need for a custom test like environment.

Hi,

I think there are a lot of good reasons to use multiple environments. Of course a staging, integration or sandbox environment should be as close as possible to production, but only close and not equal.

For instance log level, filter parameter, asset host comes into my mind. To name only a view different things. Of course you can configure all this via environment variables. But in my point of view this belongs to the application.

To configure this details via environment variables is not as clear as the environment rbs.

Furthor more I want to have different environment variable names for production environment and the rest. Simply as a kind of safety net and fast indicator when I inspect envs within my console.

Also is RAIL_ENV=staging rake …
instead of RAIL_ENV= production rake … some thing to prevent a big mistake just in case I did choose the wrong console.

I would opt for different environments. Even if it mean a huge overhead to keep this environments in sync.

To me environment variables make only sense for secret things, config options which change often or options you want to be able to change without a redeployment.

Regards Dieter

> To me environment variables make only sense for secret things

A “secret” thing should be your database connection or redis credentials. I consider this “configuration”.

> config options which change often or options you want to be able to change without a redeployment.

I’m less concerned about how often you need to change the config var, but more concerned about how sensitive it is and how bad would it be if you used the wrong thing in the wrong place.

> Also is RAIL_ENV=staging rake …

instead of RAIL_ENV= production rake … some thing to prevent a big mistake just in case I did choose the wrong console.

Most people who are running a “staging” environment will have RAILS_ENV=staging in the environment (via bash profile or config vars on Heroku). Most people would just run rake db:drop (or whatever) without specifying the RAILS_ENV. So in the case you’re in the wrong console, you’re screwed because RAILS_ENV=production would be set.

If you’re using production & staging “environments” what can happen is you’re debugging some strange behavior and you accidentally do something like export RAILS_ENV=production to see if it matches the behavior in staging, then you run rake db:drop a few seconds later then, oops you just dropped your production database. Why did your staging app need access to your production database credentials? It didn’t and should have it. If you were storing that data in the env instead of on disk via a config file you wouldn’t have hit that problem.

(granted this specific problem is extremely unlikely due to a patch I had in rails https://github.com/rails/rails/pull/22967, but the general unsafe nature of your staging app being able to modify production systems still stands, maybe it’s flushing your redisdb. You get the idea).

> I think there are a lot of good reasons to use multiple environments. Of course a staging, integration or sandbox environment should be as close as possible to production, but only close and not equal.

I think it should have the exact same behavior. The point of staging is to be able to tell if things will work in production. So if it has slightly different behavior, it’s not a reliable indicator of if production will work. I’ll give you an example.

I had a customer who had a bug with the asset pipeline in prod but not staging. It turns out have weeks of debugging they had view caching enabled in prod but not in staging which was causing an old link to an asset to be shown on production, but the bug didn’t reproduce on staging. This is an area where staging should have seen the bug but the behavior was different. Prod and staging should have the same behavior.

Where they differ should be config. They should not have the same log level, should not share the same database, should not be using the same redis instance, or should not be using the same CDN. There can be unsafe things such as charging a customer’s card, in that case stripe/paypal/whatever all have “developer” or “test” API tokens that you should be using in staging. If you’re worried about sending emails, you should be using a different account or something like a mail catcher service. I recommend env vars because that’s what’s easy on Heroku, you don’t have to store your config on env vars, but I’ve seen time and time again where having them explicitly linked to your RAILS_ENV causes lots of pain.

> Simply as a kind of safety net and fast indicator when I inspect envs within my console.

It’s a false sense of safety and actually makes it easier to mess up things. For instance I was looking at a friend’s app recently and was trying to run https://github.com/schneems/derailed_benchmarks (which I wrote). You profile your app locally and to do this accurately you need to do it with RAILS_ENV=production. We were about to hammer an endpoint when we remembered that there were some config (such as billing backend) hard coded into the app. So now we can’t run benchmarks locally, or if we try we might accidentally manipulate production data.

Even if you’re not using performance profiling sometimes bugs only show up in RAILS_ENV=production that you need to test locally. If you’re checking all your config into your app’s source that becomes a very dangerous task. If you have all your dangerous things decoupled away from RAILS_ENV then it becomes a very safe task.

RAILS_ENV is for behavior, configuration is for secrets. I’ve seen a huge number of cases where tying behavior and configuration together causes many many problems, and decoupling them solve many many problems.

Where am I coming from? I work from Heroku and have debugged literally thousands of Rails apps over the last 5 years. After writing that article and adding a warning to buildpack deploys I saw a significant drop in the number of support tickets opened on the platform. At one point is was the largest cause of customer tickets. I acknowledge separating behavior (RAILS_ENV) and config on non-heroku platforms is harder because the infrastructure for env-vars isn’t as easy, but the benefits are no different.

I’m less concerned about how often you need to change the config var, but more concerned about how sensitive it is and how bad would it be if you used the wrong thing in the wrong place.

Yes, I agree. If an information is sensitive should be the first indicator, to exclude it from your version control system. But in my point of view when you have a large old monolitc application which takes a a lot of time build, you also take into account if you want to re run your hole chain just for a change in a setting.

Most people who are running a “staging” environment will have RAILS_ENV=staging in the environment (via bash profile or config vars on Heroku). Most people would just run rake db:drop (or whatever) without specifying the RAILS_ENV. So in the case you’re in the wrong console, you’re screwed because RAILS_ENV=production would be set.

It’s like every precaution, you can use or not. I think you can compare it to sudo with password or passwordless. It’s up to you how you use it. But it’s good if it is available.

If you’re using production & staging “environments” what can happen is you’re debugging some strange behavior and you accidentally do something like export RAILS_ENV=production to see if it matches the behavior in staging, then you run rake db:drop a few seconds later then, oops you just dropped your production database.

My my fault I didn’t explain how I use it. For instance I have an env STAGING_DB… and PROD_DB … . Within my staging environment PROD_ is not set and my staging server has no connection to my production systems.

Why did your staging app need access to your production database credentials? It didn’t and should have it. If you were storing that data in the env instead of on disk via a config file you wouldn’t have hit that problem.

Your right, but a DB connection contains passwords and so on and does belong into an env. Like I said, my staging differ in behavoir for instance how it logs, what it logs.

I think it should have the exact same behavior. The point of staging is to be able to tell if things will work in production. So if it has slightly different behavior, it’s not a reliable indicator of if production will work. I’ll give you an example.

In this point I disagree a little. I agree you can experince issues with production environment you do not experince with staging, intigration or sandbox. For this it should only differ as slightly as possible. But for instance logging I want in this environments more verbose then in production. So I can nail down an issue faster.

I had a customer who had a bug with the asset pipeline in prod but not staging. It turns out have weeks of debugging they had view caching enabled in prod but not in staging which was causing an old link to an asset to be shown on production, but the bug didn’t reproduce on staging. This is an area where staging should have seen the bug but the behavior was different. Prod and staging should have the same behavior.

If you have experience with Rails, then you know that such things can happen. I did experince such kind of errors, but after a short while you try it with a “faked” production environment.

Where they differ should be config. They should not have the same log level, should not share the same database, should not be using the same redis instance, or should not be using the same CDN. There can be unsafe things such as charging a customer’s card, in that case stripe/paypal/whatever all have “developer” or “test” API tokens that you should be using in staging. If you’re worried about sending emails, you should be using a different account or something like a mail catcher service.

In my point of view when your config is mostly static and not sensitive, it belongs into an environment rb and not an environment variable. It is easier to take a look into my source code for a configuration instead of connecting to a server and check the environment variables. And least source code is controled by the developer. Environment variables are controled by the operator. This persons can be the same but must not.

It’s a false sense of safety and actually makes it easier to mess up things.

For instance I was looking at a friend’s app recently and was trying to run https://github.com/schneems/derailed_benchmarks (which I wrote). You profile your app locally and to do this accurately you need to do it with RAILS_ENV=production. We were about to hammer an endpoint when we remembered that there were some config (such as billing backend) hard coded into the app.

So now we can’t run benchmarks locally, or if we try we might accidentally manipulate production data.

Sorry, I don’t get it. This can happen with environment variables too.I don’t see the false safety.

Even if you’re not using performance profiling sometimes bugs only show up in RAILS_ENV=production that you need to test locally. If you’re checking all your config into your app’s source that becomes a very dangerous task.

Like I said. Not all config. You have to consider carefully which belongs to a config file and which to an environment variable. But that’s every time the case.

If you have all your dangerous things decoupled away from RAILS_ENV then it becomes a very safe task.

I don’t think so. You still can issue a command within a wrong console and all is set up. When you don’t export RAILS_ENV and set it with every command, then you get a tiny little more safety, as long as you don’t use copy & paste.

RAILS_ENV is for behavior, configuration is for secrets. I’ve seen a huge number of cases where tying behavior and configuration together causes many many problems, and decoupling them solve many many problems.

It’s like every thing. You have to find the right balance for your specific problem.

Where am I coming from?

I work since 2007 with Ruby and Rails. The last 5 years I did develop RoR applications in a professional environment.

I work from Heroku and have debugged literally thousands of Rails apps over the last 5 years. After writing that article and adding a warning to buildpack deploys I saw a significant drop in the number of support tickets opened on the platform. At one point is was the largest cause of customer tickets. I acknowledge separating behavior (RAILS_ENV) and config on non-heroku platforms is harder because the infrastructure for env-vars isn’t as easy, but the benefits are no different.

Heroku is in my point of view a special environment because you can only use environment variables to configure things. For instance you can’t generate config file during deployment or do other stuff.
Furthor more (realy opinionated meaning, can be realy wrong) I think most of your tickets came from RoR starters and not long term Rails users.

I think I lost you on my example of running the RAILS_ENV=production code locally & why it was safer without production config info checked in.

In my case I’m using a .env file and already have all the creds I need for local development i.e. stripe API token and AWS etc. They’re all set for my own personal development account. When I want to profile the behavior of a production app, it is super safe for me to use RAILS_ENV=production locally and be 100% confident that i’m not going to send out emails or blow away a production memcache because it’s using the exact same configuration as my development environment. When someone has all that info checked in as soon as they use RAILS_ENV=production then the app is a live loaded gun.

As you mentioned this technique doesn’t protect against the case where you ssh into your production box by accident and run a bad command. But neither does relying on someone to remember to run RAILS_ENV=staging, sure it might catch a slip up or two, but I guarantee if you rely on this, one day someone will forget that preface and tears will be made. It’s also why we should potentially build in safe guards for the really dangerous components, like dropping databases.

If you always have to remember to do something i.e. you always have to remember to preface every command with RAILS_ENV=staging, then it’s insecure by default. If you’re requiring the user to do the right thing 100% of the time, over the long run you’ll always lose. Almost all major incidents and security breaches have a human component. My desire is to mitigate this as much as possible.

> Furthor more (realy opinionated meaning, can be realy wrong) I think most of your tickets came from RoR starters and not long term Rails users.

These tickets came all types of customers. They came from some of our largest customers who have been running on the platform longer than I have worked there and new developers on free accounts. Every developer is capable of making a mistake. I’m not trying to pull rank here with my “i’ve been doing this for X long” i’m saying that i’ve seen A LOT of developers and apps make this mistake. This isn’t an idea i’m pulling out of my butt here. At one point this was the largest ticket producer by a fair margin. After I introduced the article and added a warning, they’ve mostly disappeared I’ve heard basically no complaint or pushback of the concept from people who have done this. It’s not so much an “opinion”, that can be “really wrong”, as it is a hypothesis that i’ve tested on hundreds-of-thousands/millions of rails apps and gotten a lot of positive confirmation on.

This isn’t an issue of “good” developers and “bad” developers. Here is a case of a huge outage caused by presumably someone who knows what they did yet made a mistake https://aws.amazon.com/message/41926/. The mitigation strategy is to make it so that doing the wrong thing is harder. Everyone regardless of skill or familiarity with a tool is a bad developer when they’re tired or angry or drunk. So it’s better to design for the “bad developer” case 100% of the time because it’s a reality we all face.

I built in code to explicitly protect people from dropping their production databases. Not because “only new developers do it” but because when anyone does it, regardless of how often…the results are catastrophic. Even if a problem is “easy to debug” if someone “knows what they’re doing” it’s not good enough if the problem could have been prevented via automation or code in the first place. Time is money and hours spent debugging when a issue could have been prevented entirely is, not ideal.

> For instance you can’t generate config file during deployment or do other stuff.

Tell me what you want to do on Heroku and I can tell you how it’s possible. If you needed to generate files I would hook into the rake assets:precompile call.

> For instance I have an env STAGING_DB… and PROD_DB … . Within my staging environment PROD_ is not set and my staging server has no connection to my production systems.

Great, this is what i’m talking about. Your staging or development server should not be capable of accessing any sensitive 3rd party services tied to production. If you check them into your app, then they’re accessible. You’re doing what i’m advocating here. I don’t care about how you store the information i.e. put it in an env var or some other configuration management toolchain, just whatever you do don’t put all your secrets in your app and check them in to your source control and make them available to all running instances of your app.

I think we have a common opinion in many points, but not in all. I don’t know if this is all off topic, but I like to drive this discussion a little bit further.

First we should clarify some words. This is how I understand the terms. We talked about “behavior”. It is what and how the application does things. “behavior” is defined with your
source code and can be influenced by configuration. “configuration” is every thing resided inside the config folder, environment variables and some times settings your read from other sources like
databases.

I consider now only configuration, which can be placed into an environment variable or config file. I don’t consider settings, which need to be placed within a secondary source.

“configuration” can by devided roughly into sensitive (e.g. passwords) and not sensitive parts (e.g. asset host name).

Both configurations can be devided into parts which are most likly to be changed, likly to be changed and unlikly to be changed.

I assume for my considerations that you have a build and deployment chain in place and every change to the source needs to be run through your
chain.

In this case I would put sensitive configurations always into an environment variable.

For not sensisitve configurations I would put every thing which is most likly to be changed into environment variables,
configurations which are likly to be changed, well it depends, configurations which are unlikly to be changed I put into config files
under the config folder.

 

Now comes environment files into play. Rails defines the known environments, but staging, integration and so on is also called an environment. When I follow your advice against
seperate environments within Rails then your running environments diverge from your Rails environments and every time you look at your configuration you have to make a mental shift.
If you are at development, look at development.rb, If you are at production look at production.rb, when you are at test look at your test.rb for all other environment look at production.rb.

I like the convention look at the configuration of your current environment in every case. This enables you to use ENV[‘STAGING_DB_URL’] for staging and ENV[‘PRODUCTION_DB_URL’]
within your database.yml with an “staging” section.

I think every body should use the right tool for the job and there should not be a recommendation against different environment rbs,

I think I lost you on my example of running the RAILS_ENV=production code locally & why it was safer without production config info checked in.

In my case I’m using a .env file and already have all the creds I need for local development i.e. stripe API token and AWS etc. They’re all set for my own personal development account. When I want to profile the behavior of a production app, it is super safe for me to use RAILS_ENV=production locally and be 100% confident that i’m not going to send out emails or blow away a production memcache because it’s using the exact same configuration as my development environment. When someone has all that info checked in as soon as they use RAILS_ENV=production then the app is a live loaded gun.

To me it boils down to move sensisitive configurations into environment variables. With this I totally concour. Furthor more I would use different environment variable names for production, development and so on.

As you mentioned this technique doesn’t protect against the case where you ssh into your production box by accident and run a bad command. But neither does relying on someone to remember to run RAILS_ENV=staging, sure it might catch a slip up or two, but I guarantee if you rely on this, one day someone will forget that preface and tears will be made.

Yes, some day, some one, but it will happen a little tiny bit less often as without it.

It’s also why we should potentially build in safe guards for the really dangerous components, like dropping databases.

It’s the same as with RAILS_ENV=staging. You will need to build in an option to disable this safe guard, simply because
it will be in the way of automation tasks. If you have an option in place like ‘-i’ of rm -rfi /, some people will use it, some will not. If you change the
meaning, that you have to use the switch to disable the safe guard, people will tend to use this switch every time or use an alisias
to disable the safe guard. It’s the same argumentation you use against a RAILS_ENV=staging, I think.

If you always have to remember to do something i.e. you always have to remember to preface every command with RAILS_ENV=staging, then it’s insecure by default. If you’re requiring the user to do the right thing 100% of the time, over the long run you’ll always lose. Almost all major incidents and security breaches have a human component. My desire is to mitigate this as much as possible.

If a user has to press “yes” every time, he will press “yes” also at the wrong time. So in this point your right. But it also is true for your safe guard.

These tickets came all types of customers. They came from some of our largest customers who have been running on the platform longer than I have worked there and new developers on free accounts. Every developer is capable of making a mistake. I’m not trying to pull rank here with my “i’ve been doing this for X long” i’m saying that i’ve seen A LOT of developers and apps make this mistake. This isn’t an idea i’m pulling out of my butt here. At one point this was the largest ticket producer by a fair margin. After I introduced the article and added a warning, they’ve mostly disappeared I’ve heard basically no complaint or pushback of the concept from people who have done this. It’s not so much an “opinion”, that can be “really wrong”, as it is a hypothesis that i’ve tested on hundreds-of-thousands/millions of rails apps and gotten a lot of positive confirmation on.

It depends on which part of your post you reference to. Is the test for Rails.env.staging? a bad practice? Yes, for this setting like you display it in your post it’s true. Within your post you say it “keeping your staging as close to production as possible”. This is also true. But is it a bad practice to have a staging.rb, then No.

Tell me what you want to do on Heroku and I can tell you how it’s possible. If you needed to generate files I would hook into the rake assets:precompile call.

Thanks for this offer, but I don’t use heroku. It was only an example. I use dedicated servers + chef + capistrano.

Great, this is what i’m talking about. Your staging or development server should not be capable of accessing any sensitive 3rd party services tied to production. If you check them into your app, then they’re accessible. You’re doing what i’m advocating here. I don’t care about how you store the information i.e. put it in an env var or some other configuration management toolchain, just whatever you do don’t put all your secrets in your app and check them in to your source control and make them available to all running instances of your app.

Yeah, in this point we concour.