Transformer
A neural network architecture using self-attention that became the foundation of modern LLMs after Google's 2017 'Attention Is All You Need' paper.
What Is a Transformer
A Transformer is a neural network architecture introduced in the 2017 paper “Attention Is All You Need” by Vaswani et al. at Google. It replaced recurrent and convolutional approaches for sequence modeling and became the foundation of every major LLM you use today — GPT, Claude, Gemini, Llama, Mistral. Use the term when discussing model architecture, not when describing a specific model.
How It Works
- Self-attention — each token in a sequence attends to every other token, learning context relationships in parallel rather than sequentially
- Encoder + decoder — original architecture had both; modern LLMs are usually decoder-only (GPT-style) or encoder-only (BERT-style)
- Stacked layers — typically 12–96 transformer blocks, each with multi-head attention plus a feed-forward network
- Positional encoding — added to token embeddings because attention itself is order-agnostic
Why It Matters
The Transformer’s parallelism is the reason scaling laws hold for LLMs. RNNs had to process tokens one at a time; Transformers process the whole sequence at once on GPU. That’s what made it economically feasible to train models with hundreds of billions of parameters — and it’s the bottleneck modern infrastructure (KV caches, FlashAttention) is still optimizing.
Examples
GPT-5, Claude Opus 4, Gemini 2.5 Pro, Llama 4, Mistral Large — all decoder-only Transformers. BERT and its descendants (used in older Google search) — encoder-only Transformers. Stable Diffusion’s text encoder — Transformer-based.