Chunking Strategies for RAG

Chunking is the step where you split documents into pieces before embedding them, and it quietly decides whether your RAG system works. When a production RAG system fails, retrieval is the broken layer about 73% of the time, and roughly 80% of those retrieval failures trace back to ingestion and chunking. The pragmatic 2026 default is recursive character splitting at 400 to 512 tokens with 10 to 20% overlap and token-accurate counting, then measure before you reach for anything fancier.

If you are new to the pattern, start with the Retrieval-Augmented Generation glossary entry, then come back here for the part that actually breaks in production.

What chunking is and why it decides RAG quality

A RAG pipeline indexes documents by splitting them into chunks, turning each chunk into an embedding, and storing those vectors in a vector database. At query time it retrieves the closest chunks and feeds them to the model. The chunk is the smallest unit retrieval can return, so the chunk boundary is the boundary of what your model can ever see.

That is why chunking punches above its weight. The Towards Data Science analysis behind the 73% figure found that the retrieval layer is where most RAG systems fail, and that about 80% of those retrieval failures trace back to ingestion and chunking. You can have a strong embedding model and a strong reranker and still ship a system that cannot answer basic questions, because the answer was cut in half at index time.

Fixed-size chunking: how it works, when it wins, when it shreds meaning

Fixed-size chunking cuts every N tokens (or characters) regardless of structure. It is the simplest strategy, it is fast, and it gives you predictable chunk counts and costs.

It wins when documents are flat and uniform: log lines, transcripts, plain prose with no headings or tables. It shreds meaning the moment structure matters. An MDPI Bioengineering study (November 2025) on clinical data is the clearest warning: structure-ignorant fixed-size chunking scored 13% accuracy versus 87% for adaptive topic-aligned chunking on the same corpus. A naive cut through a structured document is not a small quality hit, it is a collapse.

Use fixed-size only as a throwaway baseline, or for genuinely unstructured text where there is no structure to respect.

Recursive chunking: the pragmatic 2026 default

Recursive chunking tries to split on the largest natural boundary first (paragraphs), then falls back to smaller ones (sentences, then words) until each piece fits the size budget. It respects structure when structure exists and degrades gracefully when it does not. This is why benchmarks keep crowning it.

• Vecta and FloTorch’s February 2026 benchmark: recursive character splitting at 512 tokens hit 69% end-to-end accuracy, the highest of any strategy tested. Semantic chunking reached only about 54%.
• Chroma and Weaviate research: RecursiveCharacterTextSplitter scored 85.4 to 89.5% recall, best at 400 tokens, with the chunking method choice swinging recall by up to 9% on the same corpus.

The recommended baseline in 2026 is RecursiveCharacterTextSplitter at about 400 to 512 tokens with 10 to 20% overlap, using token-accurate counting. Start here, measure, and only move if the numbers tell you to.

Semantic chunking: embedding-based boundaries and the speed tax

Semantic chunking embeds sentences and cuts where the meaning shifts, so each chunk is topically coherent. In theory this is the right idea. In practice it carries a real cost and does not reliably beat recursive splitting.

The speed tax is steep. A Chonkie benchmark (2026) measured semantic chunking at roughly 0.33 MB/s versus about 4.82 MB/s for token-based, around 14x slower. On a large corpus that turns minutes into hours, and you re-pay it on every reindex.

The quality payoff is inconsistent. A January 2026 arXiv analysis found a “context cliff” around 2,500 tokens where answer quality drops, and found that plain sentence-based chunking matches semantic chunking up to about 5,000 tokens at a fraction of the cost. The counter-evidence: a LLMSemanticChunker reached about 0.919 recall in the Chroma and Weaviate work, so a strong semantic approach can lead on recall for some corpora. The honest read is that semantic chunking sometimes pays off, but you should prove it on your data before paying 14x.

Chunk size: matching token budget to query type

There is no universal best size, only a size that fits your query type. The general sweet spot is 128 to 512 tokens for most use cases.

Query type	Example	Chunk size	Why
Factoid / lookup	”What is the renewal fee?“	128 to 256 tokens	Small chunks isolate the fact and cut noise, so precision wins
General Q&A	”How does onboarding work?“	256 to 512 tokens	Balanced precision and context, the safe default
Analytical / summarization	”Compare the two policies”	512 to 1024+ tokens	Larger chunks keep the reasoning context together

Smaller chunks (128 to 256 tokens) win on precise factoid queries; larger chunks (512 to 1024 or more) win on analytical and summarization queries. If your traffic is mixed, default to the 256 to 512 band and let retrieval quality, not a guess, push you up or down.

Overlap: the 10-20% rule of thumb vs the “overlap may do nothing” finding

Overlap repeats a slice of text between neighboring chunks so a fact straddling a boundary still lands whole in at least one chunk. The common rule of thumb is 10 to 20% overlap.

In 2026 that rule got a useful dent. The same January 2026 arXiv analysis found overlap “provided no measurable benefit and only increased indexing cost” on its corpus. The counter-example: 15% overlap performed best on FinanceBench at 1,024-token chunks. So treat overlap as a tunable, not a mandatory default. Start at 10 to 15%, then test 0% on your own data. If it does nothing, drop it and save the indexing cost.

When-to-use comparison table

Strategy	Best document type	Best query type	Cost / speed	When to choose
Fixed-size	Flat, unstructured text (logs, transcripts)	Any, low stakes	Cheapest, fastest	A throwaway baseline, or genuinely structure-free text
Recursive	Most real documents (docs, articles, mixed)	General Q&A and factoid	Cheap, fast	Your default starting point in 2026
Semantic	Long, topic-dense prose	Analytical, topic-coherent retrieval	~14x slower, costlier	Only after recursive falls short and you can prove the gain on your data
Structure-aware (Markdown / HTML / table)	Documents with headings and tables	Anything where layout carries meaning	Moderate	Structured content where naive cuts collapse accuracy

Failure modes that wreck retrieval

These are the bugs that cost real accuracy, ranked by how often they bite:

• Broken tables. The single most common silent failure in enterprise RAG. A header like “Q2 2026 Revenue” lands in one chunk and the value “$4.2M” in another, and no embedding model or reranker can recover the row-column relationship at query time. Parse tables out and chunk them as units.
• Structure-ignorant cuts. Fixed-size splitting on structured data scored 13% accuracy versus 87% for topic-aligned chunking. Respect headings and sections.
• Severed facts and mid-sentence splits. A boundary through the middle of a claim leaves both halves useless. This is what overlap and recursive boundaries are meant to prevent.
• Character-vs-token counting bug. LangChain’s RecursiveCharacterTextSplitter counts characters by default, not tokens, so “chunk size 512” silently undershoots the real token budget unless you wire in a token-accurate length function. This is a one-line config mistake that quietly halves your effective chunks.
• The context cliff. Answer quality drops around 2,500 tokens of retrieved context. Bigger chunks are not free; past the cliff they hurt.

Going beyond plain chunks

Once the basics are solid, three techniques add retrieval quality at extra preprocessing cost:

• Contextual retrieval (Anthropic, September 2024). Prepend a short model-generated context blurb to each chunk before embedding. Contextual embeddings cut the top-20 retrieval failure rate by 35% (5.7% to 3.7%); adding contextual BM25 cut it 49%; adding reranking cut it 67% (5.7% to 1.9%), at about $1.02 per million document tokens to preprocess.
• Late chunking. Embed the whole document first, then pool per-chunk vectors so each chunk carries document-level context. On BEIR’s SciFact it lifted nDCG@10 from 64.20% to 66.10%. The catch: the document must fit the embedding model’s context window (for example 8,192 tokens for jina-embeddings-v3).
• Hierarchical (small-to-big). Retrieve on small precise chunks, then hand the model the larger parent chunk for context. This gives you factoid precision and analytical context without committing to one chunk size.

How to actually choose: a baseline-first workflow

Do not start with the clever method. Start with the boring one and earn your way up.

1. Ship the baseline. RecursiveCharacterTextSplitter, 400 to 512 tokens, 10 to 15% overlap, token-accurate counting.
2. Build an eval set. Twenty to fifty real questions with known correct source chunks. Without this you are guessing.
3. Measure retrieval, not just answers. Track recall and F1 at the retrieval layer plus end-to-end accuracy. Recall tells you if the right chunk was fetched; end-to-end tells you if the model used it.
4. Fix the failure modes first. Pull tables out, respect structure, confirm token counting. These are usually bigger wins than swapping strategies.
5. Only then test alternatives. Try semantic chunking, contextual retrieval, or different sizes one at a time against the eval set, and keep a change only if the number moves.

If you would rather not run this loop in-house, this is the exact work behind a private RAG build: getting ingestion and chunking right so retrieval stops being the layer that fails 73% of the time.