How do I prevent Rails from trying to load the application for every Rake task?

Here's what the default Rails 3 Rakefile looks like:

    require File.expand_path('../config/application', __FILE__)
    MyApp::Application.load_tasks

This does the following:

* Loads my Rails application. This takes around 15 seconds. Most of
this time is spent on the `Bundler.require`; Rails itself loads within
a second or two.
* Loads any Rake tasks from my gems.
* Loads any Rake tasks from my lib/tasks directory.

How do I write a Rakefile that only does steps 2 and 3, but not step 1
(unless I go the usual route and specify the `:environment` task)?

What would be even better is if I could explicitly require what's
needed for each gem's Rake tasks, so that if I don't want to use them,
I don't have to load them. (This was the behavior in Rails 2.)

In short, I don't want "rake foo" to try to load the Rails
environment, unless "foo" actually depends on the environment being
present.