Attention Mechanism
The core operation in Transformers that lets each token weigh the importance of every other token when computing its representation.
What Is Attention
Attention is the operation that lets a Transformer model decide which other tokens in a sequence matter when processing the current token. Instead of fixed-order processing, every token computes a weighted sum over all other tokens — the weights are the “attention” scores. Use the term when explaining how LLMs maintain context across long passages, why context windows are bounded, or why inference cost scales with sequence length.
How It Works
- Query, Key, Value — each token is projected into three vectors. The attention score is the dot product of Query with all Keys, softmaxed, then used to weight the Values
- Multi-head — the operation runs in parallel across N “heads,” each learning different relationship patterns (syntactic, semantic, positional)
- Self-attention vs cross-attention — self-attention scores within one sequence; cross-attention scores between two (used in encoder-decoder models)
- Cost — attention is O(n²) in sequence length n, which is why doubling context length quadruples compute and memory
Why It Matters
Attention is the bottleneck of LLM inference at long context. FlashAttention, sliding window attention, ring attention, and grouped-query attention all exist to reduce that O(n²) cost. If you’ve ever wondered why frontier models cap context at 200k or 1M tokens and not 100M — attention’s quadratic cost is the answer.
Examples
- Standard attention — GPT-3, original Llama
- Grouped-query attention (GQA) — Llama 3+, Mistral, reduces KV cache size
- Sliding-window attention — Mistral, attends only to a local window of recent tokens
- FlashAttention — a fused-kernel implementation that’s now standard in inference engines like vLLM and TensorRT-LLM