infrastructure
Latency
The delay between sending a request to an LLM and receiving output. In LLM serving, latency splits into time-to-first-token and tokens-per-second.
What Is Latency
Latency is how long your user waits. For LLMs it isn’t one number — it’s a pair. Time-to-first-token (TTFT) measures the gap between sending a prompt and seeing the first character of the response. Tokens-per-second (TPS) measures how fast the rest streams. A model can have low TTFT but slow TPS, or the opposite, and the user experience differs sharply. Use both metrics when comparing providers or hardware.
The Two Phases
- Prefill — the model processes the entire input prompt once. Cost scales with input length. This is what TTFT measures
- Decode — the model generates output one token at a time. Cost scales with output length. This is what TPS measures
Long input prompts hurt TTFT. Long output responses hurt total wall time.
Typical Numbers in 2026
- TTFT — 200ms to 2s for hosted frontier models, sub-100ms for small models on Groq or Cerebras
- TPS — 30 to 100 tok/s for GPT-4-class, 200 to 500 tok/s for fast providers, 1000+ tok/s on specialty hardware
- Streaming — most APIs stream tokens as they’re decoded so users see progress, but the underlying TPS is unchanged
What Increases Latency
- Large context windows (more prefill work)
- Large model size
- Long generations
- Cold starts on serverless inference
- Geographic distance to the inference endpoint
- Reasoning models that emit hidden chain-of-thought before answering
How to Reduce It
- Prompt caching for repeated system prompts
- Smaller models for routing and classification
- Streaming responses to the UI immediately
- Speculative decoding (draft model + target model)
- Picking a provider with fewer hops to your region
Latency, not accuracy, is what kills agent UX. Optimize it early.