Chunking and retrieval quality

RAG (retrieval-augmented generation) means retrieving relevant snippets of text from a knowledge base and feeding them into an LLM’s prompt as context, rather than relying solely on what the model learned during training — useful for answering questions grounded in specific, current, or proprietary documents the model was never trained on. Chunking is splitting source documents into smaller pieces, since you can’t usefully embed and retrieve an entire document as one unit, and can’t stuff arbitrarily large amounts of text into a prompt anyway. An embedding here is a numeric vector representation of a piece of text, produced by a model, positioned in a high-dimensional space such that semantically similar texts end up with similar vectors — which is what enables “find the most relevant chunks” via vector similarity search.

Chunk size is the first trade-off you hit:

• Small chunks (a sentence or two) give precise retrieval — a chunk is narrowly about one thing, so a hit is more likely to be genuinely relevant, and its embedding is not diluted by unrelated material. But each chunk carries little surrounding context, so a question whose answer needs the paragraph around it gets only a fragment, and the total number of chunks/embeddings to store and search over balloons (storage and search cost).
• Large chunks (a page) are self-contained and coherent, but the embedding becomes an average over many topics, so retrieval precision drops — a chunk can match on an irrelevant portion — and each retrieved chunk spends more of the prompt’s token budget.

Overlap — consecutive chunks sharing a boundary region, typically 10–20% — targets one specific failure: the answer landing exactly on the boundary between two chunks and being split awkwardly across both. It costs redundant storage and can cause two near-duplicate chunks to be retrieved together. Treat it as a band-aid: where a document has natural boundaries (headings, sections, functions), structural chunking that respects them is the better fix.

• Fixed-size — split every N tokens (with overlap). Trivial, format-blind, cuts mid-sentence and mid-table. A baseline, not a good answer for structured content.
• Recursive character splitting — try to split on paragraph breaks, then sentences, then words, falling back only as needed to stay under the size cap. The common default; respects prose structure cheaply. See the dedicated concept below for how the algorithm actually works.
• Structural / document-aware — split on the document’s own hierarchy: Markdown headings, HTML sections, PDF layout, code functions/classes. Best when the format is reliable; each chunk is a meaningful unit and you can attach the heading path as metadata.
• Semantic chunking — embed sentences, start a new chunk where consecutive-sentence similarity drops below a threshold (a topic shift). Adapts to content, but costs embeddings at ingest, is sensitive to the threshold, and often only marginally beats good recursive splitting.

Match the method to the corpus; a mixed corpus needs per-type handling.

It operates on an ordered list of separators, from “biggest structural break” to “no break at all.” The common default (LangChain’s RecursiveCharacterTextSplitter) uses:

["\n\n", "\n", ". ", " ", ""]

Given a chunk_size and that list:

1. Split the text on the first separator ("\n\n", paragraph breaks).
2. Walk the resulting pieces left to right, greedily merging them into a chunk until adding the next piece would exceed chunk_size; close the chunk and start a new one.
3. If a single piece is still larger than chunk_size after that split (one giant paragraph with no blank line inside it), recurse into that piece only using the next separator down the list ("\n", then ". ", then " "), repeating until the piece fits or the list is exhausted — "" is the fallback floor every recursive splitter bottoms out at: a hard character cut with no respect for structure at all.
4. Apply chunk_overlap by carrying the trailing N characters/tokens of one chunk into the start of the next, the same as fixed-size chunking.

The recursion only engages on pieces that need it — a document made of short paragraphs never falls past the first separator, so most of the text is chunked at full paragraph granularity “for free,” and only abnormally long paragraphs get forced down to sentence- or word-level splits.

Why it’s the default: no model call at ingest (pure string splitting, so it’s fast and free), deterministic, and it respects natural breaks when they exist far better than a blind every-N-tokens cut. Where it still falls short:

• Still format-blind — the separator list has no idea what a Markdown heading, an HTML table, or a code block is; "\n\n" and ". " occur inside those just as often as inside prose, so a table row or a function body can still get split mid-way. That’s what structural / document- aware chunking is for.
• The separator list must match the content — the default list is tuned for English prose. A code splitter needs language-aware separators (class/function boundaries via a parser like tree-sitter, or at minimum ["\nclass ", "\ndef ", "\n\n", "\n", " "]); CJK text needs a different sentence-delimiter assumption, since ". " is not how those languages mark sentence boundaries.
• chunk_size should be measured in the embedding model’s own tokens, not characters — sizing by raw character count under- or over-shoots the model’s real input limit, since token-per-character ratios vary by language and script.

It resolves the small-vs-large tension by decoupling the unit you search from the unit you return:

• Index and embed small chunks (precise retrieval).
• Store a link from each small chunk to a larger parent (its section, or a window of surrounding chunks).
• At query time, retrieve on the small chunks but substitute the parent into the prompt, deduped.

The model gets precise retrieval and enough context to actually answer. Variants: “sentence-window” (retrieve a sentence, expand to ±k sentences) and hierarchical indexes with summaries at each level. Cost is the extra storage and the assembly/dedup logic.

Metadata is what lets retrieval be scoped and lets the generator cite and reason about recency. Attach:

• Source identity — document id, title, URL/path, so answers can link back.
• Structural position — heading path (“Billing > Refunds > Timing”), page number, section — huge for reconstructing context and for parent lookup.
• Access control — tenant, team, visibility, so filtering enforces permissions at retrieval time, not after.
• Temporal — created/updated/effective dates, version — for “as of” filtering and for resolving conflicts between old and new chunks.
• Type — is this prose, a table, code, a changelog entry — so you can route or weight.

Then decide which fields are filterable (indexed as keywords/dates) vs merely displayed, and consider prepending key metadata (title, heading path) into the embedded text so it influences the vector.

Generic prose chunking mangles these. Options:

• Tables — do not split a table across chunks. Keep small tables whole; for large ones, chunk by row-groups but repeat the header row in each chunk. Consider generating a natural-language summary of the table to embed alongside (or instead of) the raw grid, since the raw grid embeds poorly. Very structured data may belong in a real database queried via a tool, not in RAG at all.
• Code — chunk on function/class boundaries via a parser (tree-sitter), keep imports/signatures with bodies, attach file path and language as metadata.
• Images / diagrams — run a vision model to caption or transcribe, index the text; keep a pointer to the original.
• Long lists / reference tables — often better served by exact lookup than semantic retrieval.

The meta-point: a heterogeneous corpus needs a per-content-type ingestion path, not one splitter.

Near-duplicates (boilerplate headers/footers, the same policy copied across many docs, doc versions) cause the top-k to fill with the same information repeated, crowding out other relevant material and wasting the prompt budget — and they can make a wrong-but-common phrasing dominate. Handling:

• Dedup at ingest — hash exact duplicates; for near-duplicates use MinHash/SimHash or an embedding-similarity threshold to drop or merge.
• Strip boilerplate before chunking (readability extraction, header/footer detection).
• Diversify at retrieval — Maximal Marginal Relevance (MMR) re-ranks to penalize a candidate that is too similar to already-selected results, trading a little relevance for coverage.
• Version awareness — keep only the current version in the primary index, or filter by effective date.

1. Log and inspect the retrieved chunks for the failing query. This one step resolves most cases.
2. If the answer-bearing chunk is in the retrieved set: it is a generation or context-assembly problem — lost-in-the-middle, conflicting chunks, or the prompt not forcing “answer from context”. Fix ordering, reduce k, add a re-ranker, tighten the instruction.
3. If it is not retrieved but exists in the corpus: check whether chunking split it; whether a filter excluded it; whether BM25 would have found it (vocabulary mismatch → add hybrid); embed the gold chunk and the query and look at the similarity directly.
4. If the fact is not in any chunk: it is an ingestion/coverage problem — the source was never added, or parsing dropped it.
5. Turn the fixed case into a regression test in your golden set so it cannot silently break again.