Just to be clear, what specific performance tests are you having a problem with?
Refactoring for better readability/maintainability every time your tests are green is a good practice.
Trying to make micro-optimizations in memory profile, sql queries, caching, etc without a specific failing performance test and profiling results showing which query or page load is causing the problem just burns time and obfuscates your code. Apart from anything else, the things that make your app slow are probably not those that you expect it to be.
The only thing I would say is that if you have a transient object that doesn’t require persistence (I have a location object in a new app - I don’t store locations - I just use them to handle geocoding and to return a list of places from a third party service) from what I understand (I’m still new to best Rails practices), it’s quite reasonable to put that in the lib directory. I wouldn’t wrap an entire AR lifecycle around a Class that I did not plan to persist the instances of. But that’s not to improve performance (although it might in some way) but because it seems more semantically accurate - it better expresses my intent.