Just ran into something where a gem we were using was overriding a method ActiveRecord that was the Rails 3.1/3.2 version of the method instead of the Rails 4 version.
Its tests would not have noticed any error after changing the AR dependency to 4.0 because it was only testing the modified method, not running all of the AR tests for the affected module.
The first response to this might be: “So what? People have been monkey patching Rails for what seems like forever, and they need to diff/read release notes, and be responsible for their own #$%t.” Sure, but really- how many people are going to do that? i.e. http://www.youtube.com/watch?v=8cT_Ulmcrys
It is realistic to think that maybe (like in Rails 5), the tests could be extracted for each component somehow to make it easier to run tests for one or more modules (or an entire gem), possibly allowing the ability to easily exclude specific tests. It might even be as easy as just another gemspec in the same directory with perhaps some gem metadata to indicate what the method is to run the tests (which might get a little crazy dependency-wise if you’ve got different test framework version dependencies, etc., but I doubt it is that much crazier than how things are currently).
I’m sure someone has already asked and/or gone down this road before/thought about it, but thought I’d bring it up perhaps for further discussion.
This isn’t so much a Rails problem as it is something that Rails could do to really help its community. Maybe it could even be done prior to a major version, since it wouldn’t affect Rails that much at all, functionally (I’d hope).