I'm the maintainer of an open source app that is still on 2.3.5 (http://kete.net.nz). As time progresses it definitely becomes more difficult and being behind becomes a form of technical debt, but that isn't the whole picture.
If your codebase is large, upgrades of the underlying version of Rails can be non-trivial. For quite awhile versions of 2.3.x above 2.3.5 changed pretty rapidly. It seemed to be a fairly bumpy road.
The obstacles to upgrading are specific to your code. Something that is not a big deal for others may stop you from proceeding. It really depends.
You have to weigh the value upgrading will give you (erasing some forms of technical debt, new features, etc.) vs the effort necessary to upgrade and its side effects (destabilizing your codebase).
It also hinges on resources available and priorities. Time spent upgrading that may not have benefits that are visible to a paying client is not spent on features that delivers obvious value. In other words you may have to expend more effort even justifying the work to upgrade.
At the same time, Rails has been fairly good over the long haul about providing tools for 'freezing' an application's environment. Often you can simply stick with what is already working.
So yeah, not ideal, but it's only part of the picture. Does the application give you the features you need? Is the codebase active? What's the developer community like? How's the documentation? You get the idea.