Technically, streaming uses the stream: true parameter in API calls. The server responds with a stream of SSE events, each containing one or a few tokens plus metadata (like token counts, stop reason). The client reads these events incrementally and renders them. Most SDKs handle the SSE parsing for you, but understanding the underlying mechanism helps when debugging latency issues or building custom streaming UIs.
Streaming isn't just a UI feature — it affects how you build applications. With streaming, you can't post-process the complete response before showing it (since it's not complete yet). If you need to validate, filter, or transform the response, you either process it in chunks (harder) or buffer the full response and show it after (defeating the purpose). Tools like function calling also interact with streaming: the model might stream a tool call, then pause while your code executes the tool, then resume streaming the final answer.
In a streaming context, the key latency metric is TTFT (Time to First Token) — how long before the first token appears. This depends on prompt processing time (longer prompts take longer to process before generation starts) and server load. TTFT of under 500ms feels instant; over 2 seconds feels sluggish. After the first token, inter-token latency (the gap between successive tokens) determines how smooth the stream looks. Most providers achieve 20–50ms inter-token latency, which looks natural.