Out with database.yml

rubyonrails-core
1

In short: With the help of the friendly folks in #rails-contrib, I've been working on a patch that provides Ruby configuration of your database connections. I'm looking for people to test the patch and post +1s if it works for them. You can find the patch in Lighthouse: #312 Database config in Ruby - Ruby on Rails - rails (its the last attachment, dry_database_config.diff)

## What's it look like?

In config/environment.rb:

  config.active_record.connection.configure do |db|     db.adapter = 'mysql'     db.encoding = 'utf8'   end

In config/development.rb:

config.active_record.connection.configure do |db|   db.database = 'test_app_development'   db.socket = '/tmp/mysql.sock'   db.username = 'root'   db.password = '' end

So the short of it is that you configure global stuff in environment.rb and environment-specific stuff in the respective files. This generates the connection-specification hashes that ActiveRecord::Base.establish_connection expects.

## Why change?

database.yml is workable, but its a little weird at this point. There is no other instance of YAML configuration in one's app when it runs in production. This approach also lets you significantly DRY up your configuration. Its even possible to specify the entire configuration in environment.rb.

Idatabase.yml is still loaded if you don't specify the database connection in your environment files. I wouldn't up and take it away from you like that. :slight_smile:

## Securing your database credentials

We all know its a good idea to keep your database credentials (username/password) out of source control. This patch supports that raising an exception if you try to set your username or password in a production setting. Instead, you specify a credentials file like so:

config.active_record.configure do |db|   db.database = 'test_app_production'   db.socket = '/var/run/mysql.sock'   db.credentials = "#{RAILS_ROOT}/config/credentials.rb" end

The preferred credential format is Ruby, like so:

username = 'root' password = ''

YAML is also supported:

username: root password:

## Other bits and bobs

I also patched the application generator to produce an app with no database.yml. Database config bits are added to environment.rb. The adapter-specific comments get added in environment.rb.

## Did I mention I'm looking for +1s

Yes, the pandering is strong with this one. Grab dry_database_config.diff from #312 Database config in Ruby - Ruby on Rails - rails and give it a spin.

Thanks in advance!

2

What I like with database.yml is that I don’t version it and therefore I don’t keep production passwords in version control. Also I can have one database in production locally and another on the remote machine.

3

What I like with database.yml is that I don't version it and therefore I don't keep production passwords in version control. Also I can have one database in production locally and another on the remote machine.

I've done that too. You could get that effect like so:

config.active_record.connection.configure do |db|   case `hostname`     when 'my_laptop'       db.host = 'localhost'     when 'production'       db.host = 'db1'   end end

Or you can stick with database.yml :slight_smile:

4

I will stick with yml for sure and I guess most of experienced devs will also keep their copies because, now that we have deployment set up, there is no compelling reason for us to switch.

But I’m worried what about new users? This pratice encourages them to version their passwords. Rails is opinionated and we have to choose which practice we will encourage. Will it be database info in ruby or YAML?

5

Mislav Marohnić wrote:

    Or you can stick with database.yml :slight_smile:

I will stick with yml for sure and I guess most of experienced devs will also keep their copies because, now that we have deployment set up, there is no compelling reason for us to switch.

But I'm worried what about new users? This pratice encourages them to version their passwords. Rails is opinionated and we have to choose which practice we will encourage. Will it be database info in ruby or YAML?

I don't understand how a Ruby config encourages versioning passwords more than using the yaml file. Either way you can svn/git ignore the file with the passwords.

However, the main benefit of switching to a Ruby config file seems to be "because database.yml was the only yaml config file", which doesn't seem like a particularly great reason.

On the other hand, I like this just because I find the Ruby syntax more clearly indicates what you are doing, whereas the yaml is pure data that is sucked up into some mysterious place inside Rails. So I guess that's more than a "just because it's not yaml" reason.

Ben

6

I don't think this encourages new users to version their passwords any more than they are now. In fact, because there is no username/password generated in the production config, I think it guides them towards _not_ versioning their *production* credentials. If you try to set username/password directly in your production configuration, the application won't start.

Those taking the path of least resistance will generate their app, get something small working and then check it in. When they get to the point where they want to deploy, they will have to either add a credentials file to source control (the bad route) or tweak their deploy script to symlink the credentials (the good route).

Perhaps generated applications could have more verbiage encouraging developers to store their credentials outside of source control and link it into the application at deploy-time? Besides that, there's only so much vinegar and hand-holding one can apply. :wink:

7

"No more YAML" is certainly the Reddit-friendly reason for this patch. Ousting YAML+ERB tricks is just the humane thing to do. :slight_smile:

However, the reason I really like this approach is that it cleans up clever configurations. People have been using File.exists? to figure out which MySQL socket to use within ERB blocks in database.yml for a long time. This makes those sorts of idioms easier to read and write. As the name of the patch implies, this approach is also considerably DRYer. If you want, you can define everything in environment.rb and move on.

8

That sounds like a great reason to me. Why pull in a whole different technology just to read a 6-line file?? (maybe that's what you were meaning when you said YAML was a mysterious place inside Rails?)

Or, easily work around wandering mysql socket locations (excuse the junk code):

    config.active_record.connection.configure do |db|      db.adapter = 'mysql'      sockets = %w(/var/run/mysqld/mysqld.sock /var/lib/mysql/mysql.sock /tmp/mysql.sock)      db.socket = sockets.find { |f| File.exist?(f) }    end

Does anyone even use YAML anymore? My projects are mostly JSON, a little XML, and zero YAML. database.yml is an an anachronism. Why keep this legacy inside Rails?

Great work Adam. I'll be installing your patch as I move projects to Rails 2.1 and I sure hope it moves upstream quickly.

    - Scott

9

I'm not sure how you're repeating yourself by putting the database config in a separate file. Where's the repetition?

If anything, this patch is anti-DRY because it locks up the database config in a place that can only be read by Rails. If you've got anything other than Rails that wants to talk to the database (cron jobs, other apps, whatever) YAML is a far better way to store your credentials than a chunk of Rails code (it's not even plain Ruby, because you need umpteen lines of scaffolding to evaluate it and get the values out of it). Practically anything can parse YAML, nothing except Rails can parse Rails.

I'm -1 on this patch because it'll almost certainly make my life harder trying to host apps that use this convention.

- Matt

10

Or you can configure your MySQL installation properly, and put the socket location in the global my.cnf file, as $DEITY intended.

- Matt

11

-1

I agree. We have utilities which parse database.yml for rake tasks and other batch jobs. An example (which we use on a daily basis) is a utility to automatically pull and import a production database to the local dev database, for easy testing against production data.

database.yml is metadata about an external resource. It makes sense to put that metadata in a DRY, easily-parsable format which can be used by things other than the Rails app itself.

As the previous poster mentioned, I can imagine this approach making life harder for many rails hosting providers as well. I don't think this is a good approach to encourage or propogate.

-- Chad

12

The beauty of this patch is that now the database config can be read from anywhere.

If you still want to read your config from a YAML file, nothing's stopping you. I presume you could do something like this:

    require 'yaml'     db.merge(YAML::load(File.open('database.yml')))

And it's just as easy to store your database config in XML, CSV, LDAP, or another database. Your deployment options are wide open. Really, this patch frees the database config. Where's the downside?

     - Scott

13

Because then it is nonstandard (again, making it hard for hosting companies to provide standardized auto-deployment).

Convention over configuration.

-- Chad

14
  • 1

How about we store the configuration in a c program, which rails has to compile and then execute in order to read the correct data?

What I’m trying to say is that this process doesn’t seem dryer at all. The YAML file has become the standard and many, many tutorials reference this. Do you not remember the uproar about scaffolding?

Please, for the sake of sanity, do not change this from the old way unless you have a better reason than “It’s DRYer”

15

It's different to every piece of documentation that is already out there, and it breaks Rails' motto of convention over configuration.

- Matt

16

Please, for the sake of sanity, do not change this from the old way unless you have a better reason than "It's DRYer"

I think that's a perfectly valid reason. Not just that it's more dry, but that it unifies configuration of the database along side the configuration of everything else. The special treatment for database.yml is completely unnecessary and sticks out like a wart to me these days.

This change isn't going in for 2.1.x but either for 2.2 or 3.0 (whichever the next big release will be, my guess is 3.0). And it's not going to change anything for existing applications. And it encourages the good form of separating your credentials from your configuration.

So it's all cheers from me to Adam for finally cleaning up this piece of my personal indulgence with "let's try to use YAML for something".

17

Just for the sake of argument, yaml configs don't have to be repetitive:

defaults: &defaults    adapter: mysql    encoding: utf8    username: rota    password:    host: localhost

development:    <<: *defaults    database: rota_development

test:    <<: *defaults    database: rota_test

but this is quite underused (making the default generated database.yml look like this was discussed a while back)

Fred

18

Just for the sake of argument, yaml configs don't have to be repetitive:

defaults: &defaults

That's also a lot harder to read and write, especially for newbies. I definitely support an all-ruby approach. Supporting some generic hash merging, as Scott suggested, might not be a bad idea though.

In a similar vein: last night at the Baltimore ruby group, John Trupiano suggested some way of merging the yml files from the geminstaller gem with Rails' gem configurations.

19

I've wanted to provide a non-YAML config option for GemInstaller for a while, but it is down on the priority list. And, as pointed out, if the config is in Ruby (as it is with config.gems), then it's still easy to parse a YAML file (such as geminstaller.yml) to generate that config. That's probably the best approach to support what John wanted, and would be cool to have in Rails by default (patch someone?!?)

After reading this thread, I'm convinced that both Ruby and YAML config should be supported (so I retract my -1 if that will be the case). All-ruby config is cool and nice, but for metadata describing external dependencies (Gems, Databases), there should be a STANDARDIZED, easily parsable, non-Ruby format supported as well.

So, I hope Rails still natively supports the old YAML format after this patch, to support old tutorials, and to provide a standardized option when it is needed, such as Rails parsing geminstaller.yml if it exists, or a hosting company parsing geminstaller.yml or database.yml to auto-configure whatever...

Also, I still don't think either option is DRY-er than the other, because YAML supports reuse, as was just pointed out.

-- Chad

20

Considering Rick referenced me, and I've made nary an appearance on this particular list, I figured I'd chime in. I like YAML as a portable format. It's not uncommon (especially for us) to have solutions that aren't completely ruby/rails, and where it's helpful to have portable configuration.

That said, I wouldn't be opposed to adding a config hook as suggested....just as long as that config hook allowed me to specify that I wanted my configuration read from a YAML file.

This is, in fact, the same idea Rick mentioned that I proposed. I suggested moving gem configuration out into a YAML file (or at least adding a hook for this). Why? Because then capistrano could read in that file and ensure that deployment environments have the proper versions of gems, etc. In other words, I could stop a sure-to-fail deployment before I actually re-deploy. (There's a lot more to this, but not so much core-related) I had proposed a config extension along the lines of:

config.gem_file "config/gems.yml"

This would just parse the the gems and basically iteratively run config.gem on each of them. So how is this helpful? Well my gem configuration is in one place outside of the scope of Rails. Consider the scenario where a company is managing dozens (or hundreds) of apps. You could write a simple script that would inspect all of these YAML files. This would allow you to react to new releases of major gems (oh crap, there's a new haml for instance). You could then have a handle of which apps are falling behind w.r.t. maintenance, etc.

Ultimately, I'm just reiterating the desire to have "configuration" (which is what this ugly crap is, and something convention surely can't solve) in portable formats.

I'd be happy to continue any non-core specific discussions in #rails- contrib if anyone's interested. I should be back in this afternoon.

-John