Chunking
Why it matters
Bad chunking quietly ruins RAG. If chunks are too small, they lose context: a chunk that says "this setting must be disabled" is useless if the name of the setting was in the previous chunk. If chunks are too large, the embedding becomes vague, retrieval gets less precise and you waste context window space on irrelevant text. A chunk that cuts a table or a code block in half can make both halves meaningless.
When a RAG system gives wrong answers, many teams reach for a bigger model. Often the real fix is upstream, in how documents were split.
Deep Dive
A retrieval-augmented generation (RAG) system can't usefully embed a 200-page manual as one vector. A single embedding for the whole thing would blur everything together, and even if you found it, it wouldn't fit neatly into the model's context. So before indexing, documents are cut into chunks, often a few hundred tokens each. Each chunk gets its own embedding, and at question time the system retrieves the few chunks most relevant to the query.
How you cut is one of the most important, and most underrated, decisions in a RAG build.
Common Strategies
- Fixed-size: split every N tokens or characters. Simple and predictable, but blind to structure.
- Recursive: try to split at big boundaries first (sections, then paragraphs, then sentences), falling back to smaller ones only when a piece is still too long. A sensible default for most text.
- Structure-aware: use the document's own structure, such as Markdown headings, HTML tags, PDF layout, functions in code or rows in a table.
- Semantic: embed sentences and start a new chunk wherever the topic shifts. More expensive but can produce cleaner units.
Overlap
Most setups add an overlap between neighboring chunks, often 10% to 20%, so a sentence near a boundary appears in both. This reduces the chance that a key piece of information is split across two chunks and lost.
Give Chunks Context
A chunk pulled out of a document often lacks its surroundings. Effective fixes include:
- Prepending metadata: add the document title and section heading to every chunk.
- Contextual retrieval: use an LLM to write a short note for each chunk explaining where it fits in the document, then embed the note plus the chunk. Anthropic reported in 2024 that this cut retrieval failures roughly in half, and further when combined with reranking.
- Small-to-big retrieval: search over small, precise chunks but return the larger parent section to the model.
Practical Starting Point
Start with recursive splitting at about 300 to 800 tokens with some overlap, add titles and headings to each chunk, and build a small test set of real questions with known answers. Then change one thing at a time and measure retrieval quality. The right size depends on your documents and your questions: short factual lookups favor small chunks, while questions that need explanations favor larger ones.