TTFT
Why it matters
For interactive products, TTFT is the latency users actually feel. Chat interfaces aim for a second or two at most. Voice agents are stricter: a natural conversation needs the whole pipeline of speech recognition, LLM and speech synthesis to respond in about a second, which leaves only a few hundred milliseconds for the model's first token. Code completion needs to be faster still.
For non-interactive work such as batch jobs, TTFT barely matters and total throughput is what counts. Knowing which metric your use case depends on tells you what to optimize, and which model and provider to choose.
Deep Dive
Time to first token (TTFT) measures how long a user waits before anything appears. With streaming, a response that takes 20 seconds to finish can still feel fast if the first words show up in half a second, because the user starts reading right away. With a poor TTFT, even a fast model feels sluggish.
TTFT is one of two numbers that describe LLM speed. The other is output speed, measured in tokens per second (TPS) once generation has started. Total response time is roughly TTFT plus the number of output tokens divided by TPS.
What Makes Up TTFT
- Network: the round trip to the provider, plus connection setup if the connection isn't reused.
- Queueing: waiting for capacity on a busy server. This is the part that spikes at peak hours.
- Prefill: the model processes the entire prompt before it can produce the first token. This grows with prompt length, so a 100,000-token prompt has a noticeably higher TTFT than a 1,000-token one.
- Thinking: reasoning models generate hidden reasoning tokens before the visible answer. From the user's point of view that's all waiting time, and it can stretch from seconds to minutes.
How to Reduce It
- Use prompt caching. If the long, stable part of your prompt (system prompt, documents, tool definitions) comes first and is cached, the server skips most of the prefill. Anthropic reports latency reductions of up to 85% for long prompts.
- Shorten the prompt. Retrieve fewer, better chunks and trim old history.
- Pick a smaller or faster model. Small models have lower TTFT, and specialized inference providers such as Groq and Cerebras are built for speed.
- Lower the reasoning effort, or turn off extended thinking where a quick answer is good enough.
- Stay close. Choose a region near your users and keep connections alive.
- Pay for capacity. Provisioned throughput and priority tiers reduce queueing.
Perceived Speed
When TTFT can't be reduced, manage the wait. Show progress as soon as possible: a typing indicator, a status line such as "Searching…" or a streamed summary of the model's thinking. For voice, a short acknowledgment can cover the gap. In agent products, where a full task takes minutes, the first visible sign of activity matters more than the first token of the final answer.
Measuring It Properly
- Use percentiles. An average hides the slow tail. Track p50, p95 and p99, since the slowest 5% of requests shape how the product feels.
- Measure from the client. Provider dashboards exclude network time and sometimes queueing.
- Test with your real prompts. TTFT depends on input length, so public benchmarks with short prompts can be far off for a RAG app.
- Watch it over time. TTFT changes with the provider's load through the day.
Related Metrics
Inter-token latency is the gap between consecutive tokens, which governs how smooth a stream looks. End-to-end latency is the time until the full response is done. It's the number that matters for tool calls and structured output, where nothing is usable until the last token arrives.