Out with database.yml

In short: With the help of the friendly folks in #rails-contrib, I've been working on a patch that provides Ruby configuration of your database connections. I'm looking for people to test the patch and post +1s if it works for them. You can find the patch in Lighthouse: http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/312 (its the last attachment, dry_database_config.diff)

## What's it look like?

In config/environment.rb:

  config.active_record.connection.configure do |db|
    db.adapter = 'mysql'
    db.encoding = 'utf8'
  end

In config/development.rb:

config.active_record.connection.configure do |db|
  db.database = 'test_app_development'
  db.socket = '/tmp/mysql.sock'
  db.username = 'root'
  db.password = ''
end

So the short of it is that you configure global stuff in environment.rb and environment-specific stuff in the respective files. This generates the connection-specification hashes that ActiveRecord::Base.establish_connection expects.

## Why change?

database.yml is workable, but its a little weird at this point. There is no other instance of YAML configuration in one's app when it runs in production. This approach also lets you significantly DRY up your configuration. Its even possible to specify the entire configuration in environment.rb.

Idatabase.yml is still loaded if you don't specify the database connection in your environment files. I wouldn't up and take it away from you like that. :slight_smile:

## Securing your database credentials

We all know its a good idea to keep your database credentials (username/password) out of source control. This patch supports that raising an exception if you try to set your username or password in a production setting. Instead, you specify a credentials file like so:

config.active_record.configure do |db|
  db.database = 'test_app_production'
  db.socket = '/var/run/mysql.sock'
  db.credentials = "#{RAILS_ROOT}/config/credentials.rb"
end

The preferred credential format is Ruby, like so:

username = 'root'
password = ''

YAML is also supported:

username: root
password:

## Other bits and bobs

I also patched the application generator to produce an app with no database.yml. Database config bits are added to environment.rb. The adapter-specific comments get added in environment.rb.

## Did I mention I'm looking for +1s

Yes, the pandering is strong with this one. Grab dry_database_config.diff from http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/312 and give it a spin.

Thanks in advance!

What I like with database.yml is that I don’t version it and therefore I don’t keep production passwords in version control. Also I can have one database in production locally and another on the remote machine.

What I like with database.yml is that I don't version it and therefore I don't keep production passwords in version control. Also I can have one database in production locally and another on the remote machine.

I've done that too. You could get that effect like so:

config.active_record.connection.configure do |db|
  case `hostname`
    when 'my_laptop'
      db.host = 'localhost'
    when 'production'
      db.host = 'db1'
  end
end

Or you can stick with database.yml :slight_smile:

I will stick with yml for sure and I guess most of experienced devs will also keep their copies because, now that we have deployment set up, there is no compelling reason for us to switch.

But I’m worried what about new users? This pratice encourages them to version their passwords. Rails is opinionated and we have to choose which practice we will encourage. Will it be database info in ruby or YAML?

Mislav Marohnić wrote:

    Or you can stick with database.yml :slight_smile:

I will stick with yml for sure and I guess most of experienced devs will also keep their copies because, now that we have deployment set up, there is no compelling reason for us to switch.

But I'm worried what about new users? This pratice encourages them to version their passwords. Rails is opinionated and we have to choose which practice we will encourage. Will it be database info in ruby or YAML?

I don't understand how a Ruby config encourages versioning passwords more than using the yaml file. Either way you can svn/git ignore the file with the passwords.

However, the main benefit of switching to a Ruby config file seems to be "because database.yml was the only yaml config file", which doesn't seem like a particularly great reason.

On the other hand, I like this just because I find the Ruby syntax more clearly indicates what you are doing, whereas the yaml is pure data that is sucked up into some mysterious place inside Rails. So I guess that's more than a "just because it's not yaml" reason.

Ben

I don't think this encourages new users to version their passwords any more than they are now. In fact, because there is no username/password generated in the production config, I think it guides them towards _not_ versioning their *production* credentials. If you try to set username/password directly in your production configuration, the application won't start.

Those taking the path of least resistance will generate their app, get something small working and then check it in. When they get to the point where they want to deploy, they will have to either add a credentials file to source control (the bad route) or tweak their deploy script to symlink the credentials (the good route).

Perhaps generated applications could have more verbiage encouraging developers to store their credentials outside of source control and link it into the application at deploy-time? Besides that, there's only so much vinegar and hand-holding one can apply. :wink:

"No more YAML" is certainly the Reddit-friendly reason for this patch. Ousting YAML+ERB tricks is just the humane thing to do. :slight_smile:

However, the reason I really like this approach is that it cleans up clever configurations. People have been using File.exists? to figure out which MySQL socket to use within ERB blocks in database.yml for a long time. This makes those sorts of idioms easier to read and write. As the name of the patch implies, this approach is also considerably DRYer. If you want, you can define everything in environment.rb and move on.

That sounds like a great reason to me. Why pull in a whole different
technology just to read a 6-line file?? (maybe that's what you were
meaning when you said YAML was a mysterious place inside Rails?)

Or, easily work around wandering mysql socket locations (excuse the junk code):

    config.active_record.connection.configure do |db|
     db.adapter = 'mysql'
     sockets = %w(/var/run/mysqld/mysqld.sock
/var/lib/mysql/mysql.sock /tmp/mysql.sock)
     db.socket = sockets.find { |f| File.exist?(f) }
   end

Does anyone even use YAML anymore? My projects are mostly JSON, a
little XML, and zero YAML. database.yml is an an anachronism. Why
keep this legacy inside Rails?

Great work Adam. I'll be installing your patch as I move projects to
Rails 2.1 and I sure hope it moves upstream quickly.

    - Scott

I'm not sure how you're repeating yourself by putting the database config in
a separate file. Where's the repetition?

If anything, this patch is anti-DRY because it locks up the database config
in a place that can only be read by Rails. If you've got anything other
than Rails that wants to talk to the database (cron jobs, other apps,
whatever) YAML is a far better way to store your credentials than a chunk of
Rails code (it's not even plain Ruby, because you need umpteen lines of
scaffolding to evaluate it and get the values out of it). Practically
anything can parse YAML, nothing except Rails can parse Rails.

I'm -1 on this patch because it'll almost certainly make my life harder
trying to host apps that use this convention.

- Matt

Or you can configure your MySQL installation properly, and put the socket
location in the global my.cnf file, as $DEITY intended.

- Matt

-1

I agree. We have utilities which parse database.yml for rake tasks
and other batch jobs. An example (which we use on a daily basis) is a
utility to automatically pull and import a production database to the
local dev database, for easy testing against production data.

database.yml is metadata about an external resource. It makes sense
to put that metadata in a DRY, easily-parsable format which can be
used by things other than the Rails app itself.

As the previous poster mentioned, I can imagine this approach making
life harder for many rails hosting providers as well. I don't think
this is a good approach to encourage or propogate.

-- Chad

The beauty of this patch is that now the database config can be read
from anywhere.

If you still want to read your config from a YAML file, nothing's
stopping you. I presume you could do something like this:

    require 'yaml'
    db.merge(YAML::load(File.open('database.yml')))

And it's just as easy to store your database config in XML, CSV, LDAP,
or another database. Your deployment options are wide open. Really,
this patch frees the database config. Where's the downside?

     - Scott

Because then it is nonstandard (again, making it hard for hosting
companies to provide standardized auto-deployment).

Convention over configuration.

-- Chad

  • 1

How about we store the configuration in a c program, which rails has to compile and then execute in order to read the correct data?

What I’m trying to say is that this process doesn’t seem dryer at all. The YAML file has become the standard and many, many tutorials reference this. Do you not remember the uproar about scaffolding?

Please, for the sake of sanity, do not change this from the old way unless you have a better reason than “It’s DRYer”

It's different to every piece of documentation that is already out there,
and it breaks Rails' motto of convention over configuration.

- Matt

Please, for the sake of sanity, do not change this from the old way unless
you have a better reason than "It's DRYer"

I think that's a perfectly valid reason. Not just that it's more dry,
but that it unifies configuration of the database along side the
configuration of everything else. The special treatment for
database.yml is completely unnecessary and sticks out like a wart to
me these days.

This change isn't going in for 2.1.x but either for 2.2 or 3.0
(whichever the next big release will be, my guess is 3.0). And it's
not going to change anything for existing applications. And it
encourages the good form of separating your credentials from your
configuration.

So it's all cheers from me to Adam for finally cleaning up this piece
of my personal indulgence with "let's try to use YAML for something".

Just for the sake of argument, yaml configs don't have to be repetitive:

defaults: &defaults
   adapter: mysql
   encoding: utf8
   username: rota
   password:
   host: localhost

development:
   <<: *defaults
   database: rota_development

test:
   <<: *defaults
   database: rota_test

but this is quite underused (making the default generated database.yml look like this was discussed a while back)

Fred

Just for the sake of argument, yaml configs don't have to be repetitive:

defaults: &defaults

That's also a lot harder to read and write, especially for newbies. I
definitely support an all-ruby approach. Supporting some generic hash
merging, as Scott suggested, might not be a bad idea though.

In a similar vein: last night at the Baltimore ruby group, John
Trupiano suggested some way of merging the yml files from the
geminstaller gem with Rails' gem configurations.

I've wanted to provide a non-YAML config option for GemInstaller for a
while, but it is down on the priority list. And, as pointed out, if
the config is in Ruby (as it is with config.gems), then it's still
easy to parse a YAML file (such as geminstaller.yml) to generate that
config. That's probably the best approach to support what John
wanted, and would be cool to have in Rails by default (patch
someone?!?)

After reading this thread, I'm convinced that both Ruby and YAML
config should be supported (so I retract my -1 if that will be the
case). All-ruby config is cool and nice, but for metadata describing
external dependencies (Gems, Databases), there should be a
STANDARDIZED, easily parsable, non-Ruby format supported as well.

So, I hope Rails still natively supports the old YAML format after
this patch, to support old tutorials, and to provide a standardized
option when it is needed, such as Rails parsing geminstaller.yml if it
exists, or a hosting company parsing geminstaller.yml or database.yml
to auto-configure whatever...

Also, I still don't think either option is DRY-er than the other,
because YAML supports reuse, as was just pointed out.

-- Chad

Considering Rick referenced me, and I've made nary an appearance on
this particular list, I figured I'd chime in. I like YAML as a
portable format. It's not uncommon (especially for us) to have
solutions that aren't completely ruby/rails, and where it's helpful to
have portable configuration.

That said, I wouldn't be opposed to adding a config hook as
suggested....just as long as that config hook allowed me to specify
that I wanted my configuration read from a YAML file.

This is, in fact, the same idea Rick mentioned that I proposed. I
suggested moving gem configuration out into a YAML file (or at least
adding a hook for this). Why? Because then capistrano could read in
that file and ensure that deployment environments have the proper
versions of gems, etc. In other words, I could stop a sure-to-fail
deployment before I actually re-deploy. (There's a lot more to this,
but not so much core-related) I had proposed a config extension along
the lines of:

config.gem_file "config/gems.yml"

This would just parse the the gems and basically iteratively run
config.gem on each of them. So how is this helpful? Well my gem
configuration is in one place outside of the scope of Rails. Consider
the scenario where a company is managing dozens (or hundreds) of
apps. You could write a simple script that would inspect all of these
YAML files. This would allow you to react to new releases of major
gems (oh crap, there's a new haml for instance). You could then have
a handle of which apps are falling behind w.r.t. maintenance, etc.

Ultimately, I'm just reiterating the desire to have
"configuration" (which is what this ugly crap is, and something
convention surely can't solve) in portable formats.

I'd be happy to continue any non-core specific discussions in #rails-
contrib if anyone's interested. I should be back in this afternoon.

-John