Tokenization
The process of splitting text into tokens — the numeric units an LLM actually reads, writes, and bills you for.
What Is Tokenization
Tokenization is the step that converts raw text into the integer IDs an LLM operates on. Models never see characters or words directly — they see a sequence of tokens, and every input and output is measured in tokens. Use it as the unit of accounting whenever you reason about cost, latency, or context limits.
How It Works
Modern LLMs use subword tokenizers, most commonly Byte-Pair Encoding (BPE) or SentencePiece. The tokenizer is trained on a large text corpus and learns a fixed vocabulary — typically 50K to 200K entries. Common words become single tokens. Rare words split into pieces. Whitespace, punctuation, and emoji get their own tokens.
"hello"→ 1 token"tokenization"→ 2 to 3 tokens depending on model- A typical English sentence → roughly 1 token per 0.75 words
- Code, JSON, and non-Latin scripts → more tokens per character
Common Tokenizers
- tiktoken — OpenAI’s BPE tokenizer (cl100k_base for GPT-4, o200k_base for GPT-4o)
- Claude tokenizer — Anthropic’s BPE variant, similar profile to tiktoken
- SentencePiece — Google’s tokenizer, used by Gemini and many open-weight models
- Llama tokenizer — Meta’s variant, 128K vocabulary in Llama 3
Why It Matters
Different models tokenize the same string into different counts. A 1,000-word document might be 1,300 tokens in GPT-4o and 1,500 tokens in an older model — the same content, different bill. Always measure with the target model’s tokenizer before estimating cost or context fit.