Retrieval evaluation metrics

For a query with a known, binary set of relevant documents, scored over the top-k retrieved results:

Precision@k = (relevant items in top k) / k
Recall@k    = (relevant items in top k) / (total relevant items for the query)
Hit@k       = 1 if any relevant item is in top k, else 0

• Precision@k answers “how much of what I returned is useful” — it drops if irrelevant results crowd the top k, which matters when noise directly costs something (distracts the user, wastes LLM context/tokens).
• Recall@k answers “did I find everything relevant” — it cannot decrease as k grows (a superset can only contain as many or more relevant items), so it is the natural headline metric when missing the answer is the expensive failure, as in RAG retrieval feeding a generator.
• Hit@k is the coarsest: a yes/no “was a relevant item anywhere in the top k”, used when there is essentially one right document and you only care whether it surfaced at all (e.g. “did the correct FAQ page show up”).

Precision and recall trade off against k: a larger k can only help recall and typically hurts precision, since you are pulling in results you previously left out.

A query has 3 truly relevant documents (A, B, C) out of a large corpus. The system returns the top 5 ranked as D, A, E, B, F:

Precision@5 = 2/5 = 0.40      (A and B are relevant, out of 5 returned)
Recall@5    = 2/3 = 0.67      (A and B found, out of 3 relevant total; C missed)
Hit@5       = 1               (at least one relevant item, A, is in the top 5)

Recall is capped at 0.67 no matter how these same 5 results are reordered — reordering only changes metrics that are rank-sensitive, below.

RR(query) = 1 / (rank of the first relevant item)      (0 if none appear)
MRR       = mean of RR over all queries

In the worked example above, the first relevant item (A) is at rank 2, so RR = 1/2 = 0.5. MRR only looks at the first hit — a system that puts the right document at rank 1 for half its queries and never for the other half scores the same MRR = 0.5 as one that reliably puts it at rank 2 for every query. Those are very different systems in practice; MRR alone cannot tell them apart. Use MRR when the product only ever shows or uses the first result (a “jump to this document” search box), not when several results all get used.

Precision, Recall, and Hit all treat relevance as binary, and only MRR cares about order. nDCG handles graded relevance (e.g. 0–3: irrelevant, somewhat, relevant, perfect) and rewards relevant items more the higher they rank:

DCG@k  = sum_{i=1..k}  rel_i / log2(i + 1)
IDCG@k = DCG@k computed on the ideal ordering (relevant items sorted best-first)
nDCG@k = DCG@k / IDCG@k

The log2(i + 1) discount means a hit at rank 1 counts far more than the same hit at rank 10; dividing by IDCG@k (the best possible score given that query’s relevance labels) normalizes scores to [0, 1] and makes them comparable across queries with different numbers of relevant documents — a query with 1 relevant document and a query with 20 can both score 1.0 if each achieves its own ideal ordering. This is the metric that most directly reflects lost-in-the-middle sensitivity: two systems can have identical Recall@10 while one buries its best result at rank 9 and the other puts it at rank 1 — nDCG@10 tells them apart, recall does not.

AP(query) = mean of Precision@k, taken at each rank k where a relevant item appears
MAP       = mean of AP over all queries

AP rewards a system that packs relevant items toward the top (each is “caught” at a smaller k, so that Precision@k term is larger) more than one that scatters them through the ranking. MAP is common in classic IR benchmarks (TREC) and recommendation evaluation; in RAG work it shows up less often than Recall@k/nDCG@k because it needs the full relevant set ranked rather than a fixed-size cutoff, and RAG failure analysis usually cares specifically about the fixed window of chunks actually fed to the generator.

• Feeding an LLM generator (RAG) — lead with Recall@k at the k you actually retrieve, not some larger k, since a missed document cannot be recovered downstream. Add nDCG@k if relevance is graded or lost-in-the-middle is a known risk, and Precision@k if irrelevant chunks measurably hurt (token cost, distraction).
• One “jump to it” result (search-box autocomplete, “did you mean”) — MRR.
• Ranked list the user scans (search results page, recommendations feed) — nDCG@k for graded relevance, or MAP for a single number summarizing the whole ranking rather than a fixed window.
• Report multiple k values (e.g. @5 and @20) rather than one — a system can look identical at k=5 and diverge sharply at k=20, and the right k is whatever you actually use downstream.

• k must match the real downstream window. Reporting Recall@100 for a RAG system that only feeds the top 5 chunks to the model measures a different, easier problem than the one that actually matters.
• Binary vs graded relevance labels change which metric is even valid — nDCG and MAP assume you have graded or at least ranked relevance judgments; if the golden set is just “relevant / not”, nDCG degenerates toward recall-shaped behavior and the extra complexity buys little.
• Macro vs micro averaging — averaging per-query metrics equally (macro, the normal choice) vs pooling all (query, document) pairs together (micro) gives very different numbers when query difficulty is uneven; state which you used.
• Missing labels are not “irrelevant.” A document with no relevance judgment is unlabeled, not confirmed irrelevant; an incomplete golden set silently distorts precision-like metrics by undercounting relevant items that were never labeled.
• Small eval sets produce noisy metrics — a handful of queries can swing Recall@k or nDCG by a wide margin on one lucky or unlucky query.