It all depends on your server: are you using a VPS, dedicated server, what are the server specs, how complex your application is … There simply is no magic number.
Monitoring your server load should give you an indication on when to start caching/scaling.
Caching is the first thing you should consider and Rails makes it very easy to implement and it can be done at any time. There is no real cost involved (except for your time of course). If I were you, I would consider it as soon as possible, especially if you’re on a shared host or VPS. It will make your application more performant, even when there’s no server load at all.
Peter De Berdt