KV Cache
A memory of past Key and Value attention tensors that lets a Transformer skip recomputing them on every new token during inference.
What Is a KV Cache
The KV cache is the working memory of a Transformer at inference time. When the model generates token N+1, it has already computed Key (K) and Value (V) tensors for tokens 1..N — caching them means it does not need to recompute attention over the entire sequence for every new token. Use the term when discussing inference speed, GPU memory budgets, or why long-context inference is expensive even after the prefill stage.
How It Works
- Prefill phase — the model processes the input prompt and stores K and V for each layer × head × token position
- Decode phase — each new token attends to the cached K and V; only the new token’s K and V are computed and appended
- Memory cost — KV cache size = 2 × layers × heads × head_dim × seq_len × batch_size × dtype_bytes. For a 70B model at 32k context, this is tens of GB per request
- Eviction strategies — paged attention (vLLM), prefix caching, sliding-window attention, and shared-prefix caching all exist to manage KV cache memory at scale
Why It Matters
KV cache size is the dominant memory consumer in production LLM serving — often larger than the model weights themselves. It’s why “more context” is expensive even after the prompt is processed, why batching is hard at long context, and why inference engines like vLLM, TensorRT-LLM, and SGLang exist primarily to optimize KV cache allocation. If you’ve ever wondered why prefill is fast but decode at long context is slow, the answer is KV cache bandwidth.
Examples
- vLLM — paged-attention KV cache, the standard for high-throughput serving
- TensorRT-LLM — NVIDIA’s serving engine, aggressive KV cache fusion
- SGLang — RadixAttention shares KV cache across requests with common prefixes
- llama.cpp — KV cache quantization (8-bit, 4-bit) to fit longer context on consumer hardware