Your user types a question. Hits send. Then waits. The server processes, the model generates the response word by word, but the user only sees a loading spinner for 10 seconds. Then, all at once, the full text appears. It feels like magic, but the experience is slow. The problem? You are sending the complete response instead of streaming it piece by piece. We at Meteora Web see this often in projects coming to us: developers integrating LLMs into web apps forget that the user perceives speed based on the first token, not the total time. And in a real web app, every second of wait is a potential drop-off.
This guide is for developers already familiar with LangChain and LLMs who want to bring UX to a professional level. We will implement streaming with FastAPI and Server-Sent Events (SSE), discuss costs, error handling, and optimization. If you are starting from scratch, we recommend our Pillar Guide on LangChain and LLMs.
Why does streaming improve user experience in web apps with LLMs?
The user does not care how long the model takes to generate 500 tokens. They want to see that something is happening. Streaming lowers perceived latency: the first token arrives in milliseconds instead of seconds, and the text appears progressively. UX studies show that an app displaying results in real time is perceived as 3–4 times faster, even if the total time is the same.
Sponsored Protocol
But it is not just about perception. In interactive applications (chat, assistants, collaborative editors), streaming allows the user to read and react while the model is still generating. They can stop generation if they see an error, or provide instant feedback. Additionally, it reduces server memory load: instead of holding the entire response in RAM before sending, you send it in chunks.
Concrete example: An e-commerce client had an AI assistant for sales recommendations. With full response, the user waited an average of 8 seconds. With streaming, the first recommendation appeared after 1.5 seconds. Interaction completion rate increased by 40%.
How does LLM streaming work technically?
When you call a model (ChatGPT, Claude, Llama via API), the provider exposes a stream=True parameter. Instead of returning a single JSON at the end, it sends a sequence of data: {...} chunks (SSE) or WebSocket events. Each chunk contains a fragment of text (one token, multiple tokens, or a delta). The frontend accumulates them and updates the UI in real time.
With LangChain, streaming is native. Just use stream() on the chain instead of invoke().
How to implement LLM response streaming with LangChain and FastAPI?
Let's start with a complete example. We'll use FastAPI with SSE because it is simpler than WebSocket for one-way server->client streaming. The frontend receives chunks via the EventSource API.
Sponsored Protocol
# server.py
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
import asyncio
app = FastAPI()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0, streaming=True)
async def generate_stream(prompt: str):
# LangChain stream() returns an async iterator
async for chunk in llm.astream([HumanMessage(content=prompt)]):
# chunk is an AIMessageChunk with content attribute
yield f"data: {chunk.content}\n\n"
@app.get("/chat/stream")
async def chat_stream(prompt: str, request: Request):
return StreamingResponse(
generate_stream(prompt),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no", # essential for nginx
}
)
// frontend.js
const eventSource = new EventSource(`/chat/stream?prompt=${encodeURIComponent(prompt)}`);
let responseText = "";
eventSource.onmessage = (event) => {
responseText += event.data;
document.getElementById("output").innerText = responseText;
};
eventSource.onerror = (err) => {
console.error("Stream error", err);
eventSource.close();
};
Important: If you use a reverse proxy like Nginx, you must disable buffering with the X-Accel-Buffering: no header. Otherwise Nginx accumulates chunks before sending them to the client, defeating streaming.
Sponsored Protocol
Concurrency and timeout management
In production, each user opens an SSE connection. The server must handle many simultaneous connections. FastAPI with async def uses a single event loop, so each stream is cooperative. If an LLM takes long, does it block other requests? No, because async for yields control at each chunk. However, it is wise to limit connections with a semaphore or a worker pool.
from asyncio import Semaphore
sem = Semaphore(10) # max 10 concurrent streams
async def generate_stream(prompt: str):
async with sem:
async for chunk in llm.astream(...):
yield f"data: {chunk.content}\n\n"
What are the hidden costs of LLM response streaming?
We come from accounting and we know: every token costs. Streaming does not change the number of tokens consumed, but it impacts latency and infrastructure.
- API cost: identical to a normal request. You pay for input + output tokens, regardless of streaming.
- Bandwidth cost: slightly higher due to extra SSE headers, but negligible.
- Server cost: each open SSE connection consumes resources. If you have 1000 concurrent users, you must size the server or use serverless services (e.g., Cloudflare Workers, Vercel Edge) that support streaming. We optimized a Linux server with
sysctlto increase max open connections. - Abandonment cost: if the user closes the page, the connection breaks. But the server might continue generating tokens in the background if you don't handle the
disconnectevent. FastAPI raisesasyncio.CancelledErrorwhen the client disconnects: you must catch it and stop the stream.
async def generate_stream(prompt: str, request: Request):
try:
async for chunk in llm.astream(...):
await request.is_disconnected() # check every chunk
yield f"data: {chunk.content}\n\n"
except asyncio.CancelledError:
# client disconnected, stop generation
pass
How to handle errors and interruptions during LLM streaming?
Streaming is fragile: the model may return an error mid-stream, the network connection may drop, API rate-limits may fire. The user must not see a crude error message.
Sponsored Protocol
Strategy: send a special event to signal end or error.
async def generate_stream(prompt: str):
try:
async for chunk in llm.astream(...):
yield f"data: {chunk.content}\n\n"
yield "data: [DONE]\n\n"
except Exception as e:
yield f"event: error\ndata: {str(e)}\n\n"
On the frontend, listen for the error event and show a friendly message with an option to retry.
Sponsored Protocol
Which protocol to choose: SSE or WebSocket?
It depends on the use case.
- SSE: one-way server->client communication, native over HTTP, supported by all browsers, automatically reconnects on drop. Perfect for chat and text streaming.
- WebSocket: bidirectional, more complex, useful when the client also needs to send data during streaming (e.g., interactive feedback, interruption).
For 90% of web apps with LLMs, SSE is the right choice. We built a proprietary platform to manage social media presence for multiple clients: we used SSE for real-time notifications and WebSocket only for interactive chat. Always: choose the simplest that works.
What to do now
- Test streaming on your existing project: Replace
invokewithastreamin LangChain. Measure time-to-first-token before and after. - Add disconnection handling: Check
request.is_disconnected()and catchCancelledError. Don't waste tokens. - Configure your reverse proxy: If using Nginx, add
proxy_set_header X-Accel-Buffering no;in the location block. - Monitor costs: Track token count per session. If streaming leads to longer interactions, you might have higher consumption. Consider introducing a maximum token limit per response.
- Test with real users: Perception is everything. Run an A/B test between streaming and full response. Look at completion rate and user feedback.
