Evaluating retrieval and RAG

A RAG pipeline has (at minimum) two stages that fail independently:

• Retrieval — did the system find and surface the actually-relevant source chunks for the question?
• Generation — given the retrieved context, did the LLM produce a correct, well-grounded answer from it?

If you only score the final answer end-to-end, a bad answer is ambiguous: either retrieval failed (the right context never reached the model) or generation failed (the right context was present, but the model ignored, misread, or misused it). Without separating the two, debugging is slow — teams routinely spend weeks tuning prompt wording to “fix” answers whose real cause was that chunking or the embedding model never surfaced the source material. Separated evaluation:

• Retrieval eval — given a labeled query, did the chunk(s) known to contain the answer appear in the top-k? A precision/recall-style measurement on retrieval alone that does not involve the LLM at all (see the Retrieval evaluation metrics subtopic for recall@k, MRR, nDCG and friends).
• Generation eval — given known-good context, did the model produce a faithful, complete answer? (Isolates the generator by removing retrieval as a variable.)
• End-to-end eval — the real pipeline, the number the product cares about.

• Mine real query logs — cluster and sample actual user queries so the set reflects the true distribution (head and tail), not queries you imagined.
• LLM-generated Q&A from documents — for each chunk, have an LLM write a question it answers; the chunk is the gold context. Fast way to bootstrap retrieval labels; spot-check a sample by hand and filter questions that are trivially keyword-matchable or unanswerable.
• Human review where it counts — have domain experts label a few hundred query→relevant-doc pairs for the evaluation set specifically; this is the ground truth you trust.
• Freeze it, version it, grow it — every production bug becomes a new test case. Keep the set stable enough that scores are comparable across changes.
• Watch for leakage — do not tune parameters on the same set you report final numbers on.

Aim for a few hundred queries minimum; below ~50 the metrics are too noisy to guide decisions.

For the generation stage:

• Faithfulness (groundedness) — is every claim in the answer supported by the retrieved context? Measured by decomposing the answer into atomic claims and checking each against the context (usually an LLM-as-judge or an NLI model). This is the primary hallucination metric for RAG.
• Answer relevance — does the answer actually address the question (not just say true things from the context)?
• Context precision / recall (RAGAS-style) — of the retrieved chunks, how many were used / needed; did retrieval get everything required.
• Citation accuracy — do the answer’s cited sources actually contain the cited claims.

These are mostly LLM-judged, so they inherit LLM-as-judge limitations (see the LLM topic’s Evaluating LLMs subtopic) — validate the judge against human labels on a sample, watch for length and style biases, and do not treat “the judge scored it 0.9” as verified correctness for high-stakes use.

You insert a specific fact (“the passcode is 4417”) at a known position in a long context and ask the model to retrieve it. Varying the needle’s depth and the total context length maps where recall breaks down. What it tells you: the model’s raw long-context retrieval floor, and the lost-in-the-middle shape for this model. What it does not tell you: performance on realistic tasks — real questions need synthesis across multiple facts, tolerate paraphrase rather than verbatim match, and compete with same-domain distractors rather than unrelated filler. A model can ace needle-in-a-haystack and still be unreliable on your actual multi-fact RAG queries, so treat it as a lower bound, not a proxy for quality.

• Log retrieval traces — query, retrieved chunk ids + scores, which chunks the answer cited.
• Track leading indicators — mean top-1 score, score distribution shifts (a drop often means the index went stale or an embedding-model change slipped in), rate of “I don’t know” answers, retrieval latency.
• Thumbs-down / correction signals — route negative feedback to a queue and periodically label it into the golden set.
• Shadow evaluation — replay a sample of production queries against the golden set nightly; alert on metric drops.
• Canary the index — when re-embedding or changing chunking, run the new index alongside the old on the eval set before cutting over.

The failure mode to prevent is silent drift — retrieval quality degrading over months with no error and no alarm.

Push back gently and give them a small dashboard instead of a single misleading figure, because one number hides which stage is failing and which query types are affected. A defensible summary:

• Retrieval recall@k on the golden set — “does the system find the answer”.
• End-to-end answer correctness (human-labeled sample or validated LLM-judge) — “does it answer correctly when it can”.
• Faithfulness / hallucination rate — “when it is wrong, is it confidently wrong”.
• Coverage / abstention rate — “how often it correctly says it doesn’t know”.

If forced to one number, end-to-end correctness on a representative, human-labeled query sample is the least bad choice — but report it with a confidence interval and the sample size, and revisit the sample as the query distribution shifts.