loading gem from vendor/gems in rake

Using Rails 2.1 gem dependencies, is there a way to load a gem that is
unpacked in vendor/gems from a Rake task? I have a gem that I only
want to use from Rake, so I don't want to put it in environment.rb
using config.gem.

I know you can add the gem's lib directory to the load path, but I'm
looking for something that would load the gem using the same process
as Rails does.