Context Window
The maximum number of tokens an LLM can process in a single request, including both the input prompt and the generated output.
What Is a Context Window
A context window is the working memory of an LLM — the total amount of text, measured in tokens, that the model can see at once when producing a response. It includes the system prompt, conversation history, retrieved documents, tool definitions, and the model’s own output. When you exceed the window, the oldest tokens get dropped or the request fails. Use it as your hard budget when designing any prompt, agent, or RAG pipeline.
How It Works
The window is fixed per model. Every token in your input and every token the model generates counts against the same pool. Once you hit the cap, older content is truncated. Most providers also charge separately for input and output tokens — a larger window doesn’t mean cheaper, it means more expensive.
Typical Sizes in 2025–2026
- GPT-4o / GPT-4 Turbo — 128K tokens
- Claude Sonnet 4.5 — 200K tokens (1M available in beta)
- Gemini 2.5 Pro — 1M tokens, with 2M for some tiers
- Llama 3.1 — 128K tokens
- Older GPT-3.5 — 4K to 16K tokens
Why It Matters
- Cost — large windows are billed per input token, so stuffing context is expensive
- Latency — bigger prompts mean slower first-token responses
- Recall degradation — most models lose accuracy in the middle of long contexts (“lost in the middle”)
- Design constraint — agents with long histories need summarization or RAG to stay inside the window
If your workload exceeds the window, switch to retrieval augmentation rather than buying a bigger model.