RAG architecture and the retrieval pipeline

• RAG (Retrieval-Augmented Generation) — fetching relevant text at inference time and inserting it into the prompt, so the answer is grounded in that material. Covered in full below.
• ANN (Approximate Nearest Neighbor) — a search algorithm (HNSW, IVF, …) that finds vectors very close to a query vector without scanning every vector in the index, trading a small chance of missing the true nearest match for large speed/memory gains at scale. See the embeddings-and-vector-indexes subtopic for the specific algorithms.
• BM25 — the standard lexical (keyword) ranking function: scores documents by term frequency with saturation and length normalization, no embeddings involved. See the BM25 and lexical search subtopic.

Retrieval-augmented generation is the pattern of, at inference time, fetching relevant text from an external store and inserting it into the model’s prompt as context, so the answer is grounded in that retrieved material rather than only the model’s parametric knowledge. It solves three things that neither scale nor fine-tuning address well:

• Freshness — the store can be updated continuously; the model’s weights are frozen at its training cut.
• Provenance — you can cite which documents an answer came from, and a user can verify.
• Scope and access control — proprietary or per-tenant data the model was never trained on, with retrieval filtered by the caller’s permissions.

Fine-tuning changes behavior and style, not retrievable facts; a bigger model has more parametric knowledge but still cannot know your internal wiki or today’s numbers. The corollary: RAG is the wrong tool for a behavior problem (tone, format, following conventions) — that is fine-tuning or prompting.

1. Ingestion — parse source documents (PDF, HTML, tables); parsing errors here silently corrupt everything downstream.
2. Chunking — split into retrievable units; bad boundaries strand the answer across two chunks.
3. Embedding — encode each chunk to a vector with an embedding model; a mismatched or low-quality model caps retrieval ceiling.
4. Indexing — store vectors in an ANN index (and usually the raw text + metadata in a lexical/row store); index parameters trade recall for latency.
5. Query processing — optionally rewrite, expand, or decompose the user’s query; embed it with the same model.
6. Retrieval — ANN search (and/or BM25) for top-k candidates; k too small misses, too large adds noise.
7. Re-ranking — a cross-encoder or LLM reorders candidates by true relevance; often the single highest-leverage stage.
8. Context assembly — select, order, dedupe, and format the final chunks into the prompt; lost-in-the-middle and token budget bite here.
9. Generation — the LLM answers from the assembled context; it can still ignore or misread it.
10. Post-processing — citation attribution, groundedness checking, formatting.

A weak link anywhere caps the whole system, which is why you evaluate stages independently (see the evaluation subtopic).

The user’s query and the target passages are often asymmetric — a short question (“refund policy?”) and a long policy paragraph that never uses the word “refund”. Embedding the raw query relies entirely on the embedding model bridging that gap. Transformations that help:

• Query rewriting — turn a conversational, context-dependent question (“what about the second one?”) into a standalone query using conversation history.
• Multi-query / query expansion — generate several paraphrases and retrieve for each, then merge; improves recall on vocabulary mismatch.
• HyDE (Hypothetical Document Embeddings) — ask the LLM to draft a hypothetical answer, embed that, and retrieve against it; a fake answer is distributionally closer to real passages than a question is.
• Decomposition — split a multi-part question into sub-questions, retrieve per sub-question (needed for “compare X and Y” where no single chunk covers both).
• Step-back prompting — retrieve for a more general version of the question first for background.

Cost: each adds LLM calls and latency, and expansion can pull in off-topic results, so they are applied selectively.

First-stage retrieval (ANN or BM25) optimizes for recall at low cost over the whole corpus — it uses a bi-encoder (query and document embedded independently) or term statistics, neither of which models fine-grained query–document interaction. A cross-encoder re-ranker takes the query and one candidate together through a transformer and outputs a single relevance score, so it can judge “does this passage actually answer this question” far more accurately. You run it only on the ~50–200 first-stage candidates, so the O(n) cost is bounded. Typical effect: precision@5 jumps substantially, which directly reduces both hallucination (less junk in context) and token cost (you can send fewer chunks). LLM-as-reranker is an alternative — more capable, more expensive, higher latency.

Single-shot RAG (one query → one retrieval → one answer) breaks down when:

• The question needs multiple retrieval hops (“who is the CEO of the company that acquired X” — you must find the acquirer first).
• The right query is not knowable up front — you learn what to search for from the first results.
• The answer requires aggregation across many documents (“how many incidents mentioned component Y last quarter”).

Agentic RAG gives the model a retrieval tool it calls in a loop: retrieve, read, decide whether it has enough, refine the query or ask a sub-question, retrieve again, then answer. It is strictly more capable and strictly more expensive and slower; the guards from the AI agents topic’s Tool use and agent loops subtopic (iteration caps, loop detection, budget) apply. Use it when the query distribution genuinely needs it, not by default.

• Retrieval never surfaces the answer — chunking split it, the embedding model missed the semantic match, or a metadata filter excluded it. The generator then confabulates or says “I don’t know” on a question the corpus could answer.
• The answer is present but buried — lost-in-the-middle in a long stuffed context.
• Conflicting sources — two chunks disagree (an old and a new policy) and the model picks arbitrarily or averages them.
• The model ignores the context and answers from its prior anyway — worse when the retrieved text is thin or the prompt does not instruct “answer only from the context”.
• Stale index — documents updated, embeddings not re-computed.
• No evaluation — the team cannot tell whether a change helped because there is no golden set, so tuning is vibes.

Most of these are retrieval or data problems, not model problems, and are invisible unless you instrument and evaluate the retrieval stage separately.

• Change data capture — subscribe to source updates (webhooks, a CDC stream, a crawl diff) rather than periodic full re-index.
• Content hashing — hash each chunk; re-embed and upsert only chunks whose hash changed, delete chunks whose parent document was removed. Re-embedding an unchanged corpus is pure cost.
• Deletion discipline — soft-deleted or superseded documents must actually leave the index or they keep getting retrieved; tombstones plus periodic compaction.
• Embedding-model versioning — changing the embedding model means re-embedding the entire corpus and rebuilding the index; you cannot mix vectors from two models in one space. Plan for a dual-write migration.
• Metadata as of a point in time — if answers must reflect “as of last close”, store effective dates and filter, rather than assuming the index is a live mirror.

For “less necessary”: you can now fit an entire small knowledge base in the prompt, skip the retrieval infrastructure, and let attention do the selection — simpler, no chunking artifacts, no embedding-model ceiling.

For “still necessary”: cost and latency scale with tokens, so stuffing 1M tokens on every query is economically absurd at volume; effective context is far below the advertised window (context rot, lost-in-the-middle), so recall over a full window is unreliable; corpora are routinely far larger than any window (terabytes of documents); retrieval gives provenance and access control that a big blob does not; and prompt caching only partly offsets the prefill cost.

The synthesis most practitioners land on: long context changes the granularity — retrieve fewer, larger, higher-quality chunks and lean less on precise chunk boundaries — but does not remove the need to select what goes in the prompt when the corpus is large.