rake test takes too much memory, starts swapping

I've noticed with a large rails project with lots of tests, when you run them all the amount of memory the ruby process allocates is a lot.

It seems like every object in every test is being retained. Like the garbage collector thinks objects from the very first test are still being used even when we are on test #100 and the first test's object can be released.

Is there anyway to tell rake to release objects after each test?