where is the rest of the time coming from? is this time when the cpu is
chugging away doing calculations? I don't think it would be a bandwidth
issue since I'm testing from the same router/subnet.
Yes, the remaining 16.62419 seconds are your code heating up the CPU. (Of course, not all of what falls outside of Rendering and DB is necessarily code written by you, but you'd have to instrument your code to find out more details.)
You can try New Relic's Rails Performance Monitor [1] to find out some of those details.