Embeddings Explained for RAG
What embeddings are, how they power RAG, how similarity works, and how to pick a 2026 embedding model by MTEB score, dimensions, and price.
An embedding is a list of numbers (a vector) that captures the meaning of a piece of text, so two texts that mean similar things end up close together in that number space. RAG uses embeddings to find the right documents before the model answers: you embed your text, store the vectors, then embed each question and pull back the nearest vectors. Pick a model by its retrieval score on the MTEB benchmark, its dimension count, and its price per million tokens.
What Are Embeddings
An embedding turns text into a fixed-length list of numbers. A short sentence and a 500-word page both come out as a vector of the same size, for example 1,536 numbers.
The point of those numbers is meaning. The model that produces them is trained so that text with similar meaning lands at nearby points and unrelated text lands far apart. “How do I reset my password” and “I forgot my login” produce vectors that sit close together, even though they share almost no words. “How do I reset my password” and “best pizza in Rome” sit far apart.
That is the whole trick. Once meaning is a position in space, “find related text” becomes “find nearby points,” which a computer can do fast over millions of items. Keyword search cannot do this because it matches strings, not meaning.
Why Embeddings Power RAG
Retrieval-augmented generation gives a language model facts it was not trained on by retrieving them at question time. Embeddings are the retrieval engine. The pipeline has two phases.
Indexing (done once, ahead of time):
- Chunk your documents into passages, often 200 to 800 tokens each.
- Embed every chunk with an embedding model.
- Store each vector plus its source text in a vector database.
Querying (done on every question):
- Embed the user’s question with the same model you used for the chunks.
- Search the vector database for the nearest stored vectors (the nearest-neighbor search).
- Paste the matching chunks into the prompt as context, then let the model answer.
The model never sees your whole knowledge base. It only sees the handful of chunks the embedding search pulled back, so retrieval quality decides answer quality. If you want this run over your own private data, see our private RAG service.
What Dimensions Mean
The dimension count is how many numbers are in each vector. A 256-dimension model gives you 256 numbers per text; a 3,072-dimension model gives you 3,072.
Each dimension is one axis the model can use to encode some aspect of meaning. More dimensions give the model more room to separate fine distinctions, so retrieval can be more precise. The cost is real: every extra dimension is more bytes to store and more math per search.
Doubling dimensions roughly doubles storage and slows search. Past a point the accuracy gain flattens while the bill keeps climbing, so bigger is not automatically better. For many production systems 512 to 1,024 dimensions is the sweet spot.
How Similarity Is Measured
“Nearby” needs a definition. Three metrics show up in practice.
| Metric | What it measures | When to choose |
|---|---|---|
| Cosine similarity | The angle between two vectors, ignoring length | The default. Use it unless you have a reason not to. Robust to text length differences. |
| Dot product | Angle and magnitude together | Use when your model outputs normalized vectors (then it equals cosine) or when magnitude itself carries signal. |
| Euclidean (L2) distance | Straight-line distance between the two points | Use for clustering or when absolute position matters more than direction. |
Here is the part that trips people up: when vectors are normalized to unit length (a step most modern models do for you), cosine similarity, dot product, and Euclidean distance all rank results in the same order. They are equivalent for ranking. Cosine is the default because it ignores vector length, so a long document and a short query compare on meaning alone rather than on size.
How to Pick an Embedding Model
There is no single best model. Score the candidates against your own constraints.
| Factor | What to check | Why it matters |
|---|---|---|
| MTEB retrieval score | The retrieval subset of the MTEB leaderboard, not the overall average | RAG is a retrieval task. The general average mixes in clustering and classification you do not care about. |
| Dimensions | Vector size, and whether the model supports Matryoshka truncation | Drives storage cost and search speed. Truncation lets you trade accuracy for cost later. |
| Max context length | The token limit per input | If your chunks exceed it, the model silently truncates and you lose the tail. |
| Multilingual / multimodal | Languages covered, and whether it embeds images too | Matters if your corpus is not English-only or includes screenshots and PDFs. |
| Price | Cost per million tokens (API) or hardware you can run (self-host) | Embedding a large corpus is a one-time bill that can surprise you. |
| Self-host vs API | Whether you can run open weights locally | Data residency, privacy, and per-token cost at scale. |
A good default for most teams: start with an API model that scores well on MTEB retrieval, pick the smallest dimension count that still hits your accuracy bar, and only move to self-hosting once volume makes the API bill hurt.
2026 Embedding Model Comparison
Figures are approximate and change often. Always confirm current dimensions, limits, and price on the provider’s own page before committing.
| Model | Dimensions | MTEB retrieval (approx) | Price / 1M tokens | When to choose |
|---|---|---|---|---|
| OpenAI text-embedding-3-small | 1,536 (truncatable) | ~62 | ~$0.02 | Cheapest sane default, easy if you already use OpenAI. |
| OpenAI text-embedding-3-large | 3,072 (truncatable) | ~65 | ~$0.13 | Higher accuracy on the same stack when small is not enough. |
| Google gemini-embedding-001 | up to 3,072 (Matryoshka) | ~68 | ~$0.15 | Top-tier accuracy and flexible truncation inside Google Cloud. |
| Voyage voyage-3-large | 1,024 (Matryoshka) | ~70 | ~$0.18 | Best-in-class retrieval, strong on long documents and code. |
| Cohere embed-v4 | 1,536 | ~67 | ~$0.12 | Strong multilingual and native multimodal (text plus images). |
| Jina embeddings v3 | 1,024 (Matryoshka) | ~65 | ~$0.02 / self-host | Open weights, long 8K context, cheap or free if you self-host. |
Pair any of these with a vector store such as Pinecone or an open-source option you run yourself.
What Embeddings Cost
Embeddings hit your bill in two separate places, and people usually forget the second one.
Line 1, embedding the text (tokens). You pay per million tokens to turn text into vectors. This is large at first (you embed the whole corpus once) and smaller after (only new or changed documents, plus every query).
Line 2, storing the vectors (ongoing). The vector database charges for the storage and the search capacity, and that bill never stops while the data is live.
Worked example. Say you have 1 million chunks averaging 500 tokens, embedded with text-embedding-3-small at ~$0.02 per million tokens:
- Tokens to embed: 1,000,000 chunks x 500 tokens = 500M tokens.
- One-time embedding cost: 500M / 1M x $0.02 = $10. Embedding a million chunks is cheap.
- Storage: 1M vectors x 1,536 dims x 4 bytes (float32) = ~6.1 GB of raw vectors, before index overhead. That GB-scale storage, replicated and indexed, is what you pay for month after month.
The lesson: the one-time embedding run is rarely the problem. Ongoing storage and dimension count are where the recurring cost lives.
Cutting Cost: Dimensions and Quantization
Two levers cut the storage line without re-architecting anything.
Matryoshka truncation. Models like text-embedding-3, gemini-embedding-001, and voyage-3 are trained so the most important information sits in the early dimensions. You can chop a 1,536-dim vector down to 512 and keep most of the accuracy. Going from 1,536 to 512 dimensions cuts storage by about two-thirds. Test the accuracy drop on your own data first; it is usually small.
Quantization. Store each number in fewer bits.
| Precision | Bytes per dimension | Storage vs float32 | Accuracy impact |
|---|---|---|---|
| float32 | 4 | baseline (100%) | none |
| int8 | 1 | ~25% (4x smaller) | small, often negligible |
| binary | 0.125 | ~3% (32x smaller) | larger, recover with reranking |
Take the 6.1 GB example above. int8 quantization brings it to ~1.5 GB. Combine int8 with truncation to 512 dimensions and you are near 0.5 GB, a ~12x cut, for a small and measurable accuracy cost. See the quantization glossary entry for the underlying idea.
Common Mistakes
- Mismatched query and document models. You must embed queries with the exact same model that embedded the documents. Different models put meaning in different places, so cross-model vectors do not compare. This silently wrecks retrieval.
- Re-embedding everything on every change. You only re-embed documents that actually changed, not the whole corpus, unless you switched models. Track which chunks are dirty.
- Ignoring max token limits. Chunks longer than the model’s context get truncated, and the cut-off text is just lost from the vector. Chunk below the limit on purpose.
- Over-paying for dimensions you do not need. Defaulting to the largest model and full dimensions when 512 dims would have hit your accuracy bar means you pay for storage and search you never use.
FAQ
How are embeddings different from LLMs? An LLM reads text and generates text. An embedding model reads text and outputs a vector of numbers, with no generation at all. They are different tools: the embedding model finds the right context, the LLM writes the answer. RAG uses both.
Can I mix embedding models? Not within one index. Every vector you compare has to come from the same model. You can run separate indexes (one per model) and search them independently, but you cannot put vectors from two models in one space and compare them.
Do I re-embed when I switch models? Yes, fully. A new model produces an incompatible vector space, so the entire corpus has to be re-embedded and the old index discarded. Budget the one-time token cost before you switch.
How many dimensions do I need? Start at 512 to 1,024 for most RAG systems. Test retrieval accuracy on your own questions, then raise dimensions only if the numbers say you need them. More dimensions is more cost, not free precision.
Next Steps
- Refresh the core idea in the embedding glossary entry.
- Understand the storage layer in vector database.
- For a managed store, review Pinecone in our services catalog.
- To run retrieval over your own private documents, see the private RAG service.