Quick set of suggestions:
1) Make sure that you aren't storing the values in a @@ class variable or similar long lived area.
2) Don't store these objects in the session (or any objects other than base types). The session is not faster than the database you're already using and it creates copies.
3) Run a multiuser test against your application and see if it's making many copies of the same data. If so then either use or write a cache library for it that uses weakref so they can be collected, but share the data between all the clients. This is *really* hard to get right.
4) Mongrel has a simple -B option which will log memory usage to log/mongrel_debug/objects.log and might help find out what's going on.
Hope that helps.