I spent a while getting token-by-token streaming working from a Rails 8 app over server-sent events, and four things cost me real time. None of them were obvious from the docs, and all of them share a nasty property: the code is correct and the symptom looks like a completely different problem. Writing them down in case it saves someone an afternoon.
1. X-Accel-Buffering: no
If there is a reverse proxy in front of your app, it will cheerfully buffer your entire “stream” and hand it over as one lump at the end. Locally it works perfectly. In production it silently stops being a stream, and nothing in your logs is wrong, because nothing on the Rails side is wrong.
response.headers["Content-Type"] = "text/event-stream"
response.headers["Cache-Control"] = "no-cache"
response.headers["X-Accel-Buffering"] = "no"
The third one is the one people miss. Worth setting even if you don’t think nginx is in your path, because something nginx-shaped usually is.
2. Providers don’t stream one token at a time
I assumed a delta stream was roughly one token per chunk. Some providers batch — you get a chunk with a dozen words in it, a pause, then another batch. Pipe those deltas straight to the browser and the output arrives in visible lumps. It reads as broken even though it is byte-for-byte what the provider sent.
The fix is to decouple the receive rate from the emit rate: buffer what arrives and emit it word by word on a short timer, so the reader sees steady motion. It is purely cosmetic and it makes an enormous difference to whether the feature feels finished or feels like a bug.
3. A client disconnecting is normal, not exceptional
Someone closes the tab mid-answer and your next response.stream.write raises. That is the ordinary path, not an error condition, and if you don’t handle it you will drown in exception noise generated by people using your app correctly.
def create
# ... write frames ...
rescue ActionController::Live::ClientDisconnected, IOError
# normal: the reader went away
ensure
response.stream.close
end
The ensure matters on its own account. A live response that never gets closed holds its thread open.
4. ActionController::Live occupies a thread for the whole request
Each streaming response holds a server thread for its entire lifetime, and for an LLM answer that can be twenty or thirty seconds. On Puma that pool is fixed, so a fairly modest number of concurrent readers can starve the rest of the app — and it presents as unrelated endpoints getting slow, which sends you looking at your database instead of at this. Worth knowing before you meet it under load rather than after.
If you don’t specifically need SSE, Turbo Streams over Action Cable moves this problem onto the Cable server instead. Different tradeoffs, but not this one.
The through-line for all four is that the failure mode disguises itself: a proxy problem looks like your code, a batching provider looks like a rendering bug, and thread starvation looks like a slow query. Hope it helps someone.