Evaluating LLMs: benchmarks, contamination, judges

• Benchmark — a fixed set of test questions with known answers, used to compare models by a single score.
• Eval (evaluation) — any repeatable procedure that turns model outputs into a number or a pass/fail; a benchmark is a public, generic eval.
• Test set / held-out set — examples the model was not trained on, so the score reflects generalisation rather than memorisation.
• Contamination — the test set (or its answers) appearing in the model’s training data, inflating the score.
• Memorisation — reproducing training text near-verbatim instead of solving the task from understanding.
• Canary string — a unique random string embedded in a benchmark file so that anyone can check whether a model has seen that file.
• Accuracy — the fraction of items answered correctly, correct / total.
• pass@k — the probability that at least one of k sampled attempts passes the checker (usually unit tests for code).
• Sampling / temperature — drawing a random completion from the model’s output distribution; higher temperature gives more varied draws.
• Unbiased estimator — a formula whose average over many repetitions equals the true quantity; a biased one is systematically too high or too low.
• Binomial coefficient C(a, b) — “a choose b”: the number of ways to pick b items from a, ignoring order. C(5, 2) = 10.
• Log-probability (logprob) — the natural logarithm of the probability the model assigned to a token; always <= 0.
• Cross-entropy — the average negative log-probability the model gives to the true next tokens; lower is better.
• Perplexity (PPL) — exp(cross-entropy); read as “the model is as unsure as if it were choosing uniformly among PPL options at each step”.
• Standard error (SE) — the typical size of the random noise in a measured quantity such as an accuracy.
• Confidence interval (CI) — a range that, over repeated experiments, contains the true value a stated fraction (usually 95%) of the time.
• Wilson interval — a better-behaved CI for proportions than the textbook one, especially for small samples or extreme accuracies.
• Paired comparison — evaluating two systems on the same items so item difficulty cancels out of the difference.
• Elo rating — a number per player such that the difference between two ratings predicts the win probability in a head-to-head match.
• LLM-as-judge — using a (usually stronger) LLM to grade or rank other models’ outputs against a rubric.
• Positional bias — a judge’s tendency to prefer the answer shown first (or second) regardless of quality.
• Cohen’s kappa (kappa) — agreement between two raters corrected for the agreement expected by chance; 0 = chance, 1 = perfect.
• Regression (in testing) — a change that makes something that used to work stop working.
• Rubric — an explicit list of criteria with scoring rules that a judge (human or LLM) applies.
• Saturation — a benchmark where top models all score near the maximum, so it no longer separates them.
• Few-shot / 0-shot — including a few worked examples in the prompt / including none.
• Answer extraction — the rule (regex, first letter, judge) that turns free text into the answer that gets graded.
• Prompt template — a prompt stored as text with named placeholders that are filled in at call time, rather than built ad hoc in code.
• f-string — Python’s inline string interpolation (f"Summarise {doc}"); the typical way prompts end up scattered through code.
• Regression test suite — a fixed set of inputs with assertions on the outputs, re-run on every change to catch regressions.

Contamination is when a model’s training data has overlapped with a benchmark’s test set, so a high score reflects memorisation rather than the capability the benchmark claims to measure. A student who has seen the exam paper beforehand and a student who understands the subject can score the same; the score alone cannot tell them apart.

How it happens without anyone cheating

1. Direct leak — benchmark questions and answers are posted publicly (GitHub repos, forum threads, blog posts walking through the solutions) and a web crawl swallows them into the pretraining corpus.
2. Shared source — the benchmark reuses or lightly paraphrases material from textbooks, Wikipedia or exam archives that are also in the training data.
3. Second-hand leak — a later model’s training data contains pages that quote an earlier model’s benchmark outputs, so answers propagate indirectly.
4. Tuning leak — developers iterate on a benchmark during development (picking the checkpoint or prompt that scores best on it); nothing is memorised, but the number is now optimised for that exact set, which is its own kind of overfitting.

Detecting it

Method	What you do	What it tells you
n-gram overlap	Search training data (where you have access) for long word sequences (e.g. 8 to 13 tokens) that also appear in test items	Direct textual leakage
Canary string	Many benchmark files embed a unique GUID; ask the model to complete or recognise it	Whether the benchmark file was scraped
Perturbed twin	Rewrite items (change numbers, rename entities, reorder options) and re-score	A big drop on twins suggests the original score leaned on memory
Fresh held-out set	Write new items of the same kind after the model’s training cut-off	The cleanest test; costs effort
Verbatim completion	Give the first half of an item, check if the model reproduces the second half exactly	Memorisation of that specific item

Worked illustration (numbers are illustrative)

A model scores 82% on a public benchmark. You hand-write 100 new questions of the same style, with fresh numbers and names, and it scores 64%. The 18-point gap is far larger than the roughly +/-8 points of noise a 100-item set carries (see Confidence intervals on accuracy), so the original score was probably inflated by exposure. The converse is not proof: equal scores do not show a model is clean, only that you found no evidence.

Gotchas

• Absence of n-gram matches is weak evidence: paraphrased leakage and translated leakage escape exact matching.
• Contamination is a spectrum (question only, question plus answer, whole file), and each level inflates the score by a different amount.
• Vendors usually cannot publish their training data, so outside readers often cannot check at all; treat old public benchmarks with suspicion in proportion to their age and fame.

A benchmark score is one number summarising a model on a fixed sample of tasks, in one fixed prompt format, graded in one fixed way. Two models can match on that number while differing on everything the number does not capture. Real usage is a broader, differently shaped distribution.

Six reasons the same score hides different behaviour

• Prompt-format sensitivity — a model evaluated under a particular few-shot template can score well on that exact format and worse under your application’s prompting conventions. Re-running the same benchmark with only the answer-option labels changed (A/B/C/D to 1/2/3/4) can move scores by points.
• Style and length bias in scoring — many benchmarks, and especially LLM-judged ones, correlate with answer length and confident phrasing more than with correctness.
• Narrow task coverage — a benchmark heavy on competition mathematics says little about tool use, following a long system prompt or multi-turn coherence, all of which matter more for a typical product.
• Aggregate hides variance — see the table below.
• Saturation — when top models all score 95% or more, the remaining differences are mostly label noise and ambiguous items, not capability.
• Sampling noise — a score on a few hundred items carries an error bar (see Confidence intervals on accuracy); a 1-point gap between two models is usually inside it.

What a benchmark score is made of

reported score = f( model weights,
                    prompt template and few-shot examples,
                    decoding settings (temperature, max_tokens),
                    answer extraction rule (regex? first letter? judge?),
                    which items were chosen,
                    contamination )

Change any argument and the result changes while the model is “the same”. Two published numbers for the “same” model and benchmark can differ because one lab used 5-shot with chain-of-thought and another used 0-shot with a strict regex for the final answer.

Aggregate scores hide the part you care about (illustrative)

Category (20 items each)	Model A	Model B
Arithmetic	100%	70%
Summarisation	100%	70%
Table extraction	100%	70%
Legal clause lookup	20%	70%
Mean of the four	80%	70%

Model A wins on the headline number, yet if your whole product is legal clause lookup, Model B is more than three times better for you (70% vs 20%). The headline is a weighted average, and the benchmark’s weights are not your weights.

Same mean, different shape (illustrative)

Two models score 75% on a 100-item benchmark. Break the items into four equal groups:

Group (25 items)	Model C	Model D
Short questions	24 of 25	19 of 25
Long documents	12 of 25	19 of 25
Multi-step	24 of 25	19 of 25
Rare formats	15 of 25	18 of 25
Total	75	75

Model C is excellent on short and multi-step items but weak on long documents; Model D is uniformly mediocre. A product that feeds long documents will behave very differently on the two, though both “scored 75”.

Code: per-category scores and a paired comparison

from collections import defaultdict
from math import sqrt

# (category, model_a_correct, model_b_correct) per item; illustrative toy data
results = (
    [("short", 1, 1)] * 20 + [("short", 1, 0)] * 4 + [("short", 0, 1)] * 1 +
    [("long", 0, 1)] * 10 + [("long", 1, 1)] * 5 + [("long", 0, 0)] * 10
)

per_cat = defaultdict(lambda: [0, 0, 0])
for cat, a, b in results:
    per_cat[cat][0] += a
    per_cat[cat][1] += b
    per_cat[cat][2] += 1
for cat, (a, b, n) in per_cat.items():
    print(f"{cat:6s} A={a/n:.2f}  B={b/n:.2f}  (n={n})")
# short  A=0.96  B=0.84  (n=25)
# long   A=0.20  B=0.60  (n=25)

n = len(results)
b = sum(1 for _, a, bb in results if a and not bb)     # A right, B wrong
c = sum(1 for _, a, bb in results if bb and not a)     # B right, A wrong
print(f"overall A={sum(r[1] for r in results)/n:.2f} B={sum(r[2] for r in results)/n:.2f}")
print(f"disagreements: A-only={b}, B-only={c}, z={(b - c) / sqrt(b + c):.2f}")
# overall A=0.58 B=0.72
# disagreements: A-only=4, B-only=11, z=-1.81

Walkthrough: the loop builds a score per category rather than one mean, which is what reveals that A is far better on short items and far worse on long ones. The last block is the paired test from Confidence intervals on accuracy: only the 15 items where the models disagree carry information, and z = -1.81 says B’s lead is suggestive but not conclusive at the 5% level. Both pieces of information (per-category and paired) are invisible in a single mean.

What to do instead

1. Build or sample an eval from your inputs and format (see Building a task-specific eval from scratch).
2. Report per-category scores and the number of items in each.
3. Run the public benchmark’s prompt format and your own, and compare; a large gap is a warning.
4. Read a sample of the failures by hand. Two models with the same score fail differently.
5. Treat differences smaller than the confidence interval as ties.

pass@k is the probability that at least one of k sampled completions passes the checker (typically unit tests). It matters where you can verify answers automatically and where retrying is cheap: code generation, maths with a verifiable answer, agents with a test suite.

The naive estimates, and what is wrong with them

Suppose the true chance that a single sample passes is p. Then the true value is pass@k = 1 - (1 - p)^k (the chance that all k fail is (1-p)^k). You never know p, so you estimate it. There are two naive routes:

1. Draw exactly k samples, check if any pass. Each problem gives only a 0/1 outcome, so the estimate is extremely noisy, and you need a fresh run for every k you want to report.
2. Estimate p_hat = c / n and plug in 1 - (1 - p_hat)^k. This is smooth, but biased: (1 - p)^k is a convex function, so by Jensen’s inequality the plug-in value of (1 - p_hat)^k is on average too high, which makes the pass@k estimate too low.

The standard unbiased estimator

Sample n >= k completions per problem, count c of them that pass, and compute:

pass@k = 1 - C(n - c, k) / C(n, k)

Symbol	Meaning	Where it comes from
n	total completions you sampled for this problem	you choose it (e.g. 10 or 200); bigger is more precise and more expensive
c	how many of those n passed the checker	measured by running the tests
k	the attempt budget you want to report (1, 5, 10, …)	you choose it; must be <= n
C(a, b)	binomial coefficient, “a choose b” = a! / (b! (a-b)!)	combinatorics
C(n, k)	the number of distinct ways to pick k of the n samples	counting
C(n - c, k)	the number of those picks that contain only failures	counting

In words: if you draw k of your n samples at random without replacement, C(n-c, k) / C(n, k) is the fraction of draws that are all failures, so one minus that is the fraction with at least one success. Averaged over the random draw of samples, this is an unbiased estimate of 1 - (1-p)^k.

Worked example: n = 10, c = 3

For k = 5: C(7, 5) = 21 all-fail subsets out of C(10, 5) = 252. So pass@5 = 1 - 21/252 = 0.9167. The plug-in estimate gives 1 - (1 - 0.3)^5 = 0.832, noticeably lower.

k	unbiased 1 - C(7,k)/C(10,k)	plug-in 1 - 0.7^k
1	0.300	0.300
2	0.533	0.510
5	0.917	0.832
10	1.000	0.972

At k = 10 = n every sample is used, and since 3 passed, at least one is certainly in the pick, so pass@10 = 1 exactly; the plug-in can never reach it.

Code

from math import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    """Direct form. Fine for small n."""
    if n - c < k:          # fewer failures than picks -> a pass is guaranteed
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

def pass_at_k_stable(n: int, c: int, k: int) -> float:
    """Product form: avoids huge factorials when n is in the hundreds."""
    if n - c < k:
        return 1.0
    prod = 1.0
    for i in range(n - c + 1, n + 1):
        prod *= 1.0 - k / i
    return 1.0 - prod

print([round(pass_at_k(10, 3, k), 4) for k in (1, 2, 5, 10)])
# -> [0.3, 0.5333, 0.9167, 1.0]
print([round(pass_at_k_stable(10, 3, k), 4) for k in (1, 2, 5, 10)])
# -> [0.3, 0.5333, 0.9167, 1.0]
print(round(pass_at_k_stable(200, 20, 1), 3),    # 0.1   (c/n = 10%)
      round(pass_at_k_stable(200, 20, 10), 3),   # 0.660
      round(pass_at_k_stable(200, 20, 100), 3))  # 1.0

Walkthrough: the first function is the formula verbatim. The second uses the identity C(n-c, k) / C(n, k) = prod_{i=n-c+1..n} (1 - k/i), which multiplies c numbers between 0 and 1 and so never overflows. The last line shows a model that passes 10% of single attempts reaches 66% with 10 attempts and essentially 100% with 100.

Averaging and gotchas

• Compute pass@k per problem, then average across problems. Do not pool all samples first.
• pass@k for large k rewards a model that is right occasionally. It is only meaningful as a product metric if you also have a cheap, reliable verifier that can pick the passing sample, otherwise a user sees pass@1.
• n samples must be drawn at the same temperature you intend to report; greedy decoding makes all n identical and the estimator meaningless.

Perplexity measures how well a language model predicts a text. It is used for pretraining progress, comparing tokenizer-matched models, and checking that quantisation or pruning did not damage a model. It is not a measure of whether answers are correct or helpful.

The formula, symbol by symbol

cross-entropy  H = -(1/N) * sum_{t=1..N} ln p(x_t | x_1 ... x_{t-1})
perplexity   PPL = exp(H)

Symbol	Meaning	Where it comes from
N	number of tokens in the evaluation text	counted after tokenising
x_t	the actual token at position t	the reference text (ground truth)
x_1 ... x_{t-1}	all earlier tokens (the context)	the same text
`p(x_t	…)`	the probability the model gave to the true next token
ln	natural logarithm	converts probabilities to additive “surprise”
-ln p	surprise of one token: 0 if certain, large if the model thought it unlikely	derived
exp(H)	undoes the log, giving an “effective number of equally likely choices”	derived

Worked example

A 4-token sentence where the model gave the true tokens these probabilities: 0.5, 0.25, 0.1, 0.8.

-ln 0.5  = 0.693      -ln 0.25 = 1.386      -ln 0.1 = 2.303      -ln 0.8 = 0.223
H   = (0.693 + 1.386 + 2.303 + 0.223) / 4 = 1.151
PPL = exp(1.151) = 3.16

Reading: the model was, on average, as uncertain as if it were picking uniformly among about 3.2 options per token. Two sanity anchors: a model that always assigns the true token probability 1 has PPL = 1; a model that spreads probability uniformly over a 50,000-token vocabulary has PPL = 50,000 (H = ln 50,000 = 10.82).

from math import log, exp

true_token_probs = [0.5, 0.25, 0.1, 0.8]
h = -sum(log(p) for p in true_token_probs) / len(true_token_probs)
print(round(h, 3), round(exp(h), 2))   # -> 1.151 3.16

Pitfalls

• Not comparable across tokenizers. A model with a bigger vocabulary has fewer tokens per document, so its per-token perplexity is not on the same scale. Compare per-byte or per-character measures (bits per byte) across different tokenizers.
• Depends on the text. Perplexity on code, on legal prose and on chat differ greatly; compare models on the same held-out text only.
• Contamination applies. If the eval text was in training, perplexity drops without any generalisation gain.
• Instruction-tuned models often have worse perplexity on web text than their base versions while being much more useful, so perplexity is the wrong yardstick for chat quality.

A benchmark score is an estimate from a finite sample. If you re-drew a different set of 20 questions the score would change even for the same model. A confidence interval quantifies that noise.

The formulas

For k correct out of n items, accuracy p_hat = k / n. The textbook 95% interval is:

SE = sqrt( p_hat * (1 - p_hat) / n )          CI = p_hat +/- 1.96 * SE

Symbol	Meaning	Where it comes from
n	number of test items	the size of your eval set
k	items answered correctly	measured
p_hat	observed accuracy	k / n
SE	standard error: the spread of p_hat under re-sampling	the variance of a yes/no variable is p(1-p); averaging n of them divides variance by n
1.96	z-value covering the central 95% of a normal curve	a constant of the normal distribution; chosen by the 95% level

The sqrt(n) in the denominator is the important part: to halve the interval you need four times the data.

Worked numbers (accuracy 70% in every row)

n	k	SE	normal 95% CI	Wilson 95% CI
20	14	0.102	0.50 to 0.90	0.48 to 0.85
100	70	0.046	0.61 to 0.79	0.60 to 0.78
1,000	700	0.0145	0.672 to 0.728	0.671 to 0.728
10,000	7,000	0.0046	0.691 to 0.709	0.691 to 0.709

So on 20 examples, “70%” is compatible with anything from about 50% to 90%: a 3-point change is pure noise. To measure the accuracy to within +/-3 points at 95% confidence you need roughly n = 1.96^2 * 0.7 * 0.3 / 0.03^2 = 896 items.

from math import sqrt

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    p = k / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return centre - half, centre + half

print([round(x, 3) for x in wilson(14, 20)])      # -> [0.481, 0.855]
print([round(x, 3) for x in wilson(700, 1000)])   # -> [0.671, 0.728]

The Wilson interval is preferable to the textbook one because it stays inside 0 to 1 and behaves for small n or accuracies near 0% or 100% (the normal interval can run past 100%).

Comparing two models: use a paired test

Two models evaluated on the same 200 items have separate intervals that overlap a lot, but that is the wrong test, because the same hard items drag both down. Count only the items where they disagree:

• b = items model A got right and B got wrong; c = items B got right and A got wrong.
• Items both got right (or both wrong) carry no information about which is better.

Example: n = 200, b = 20, c = 10  ->  A is 10 items better = +5.0 points
z = (b - c) / sqrt(b + c) = 10 / sqrt(30) = 1.83   (about p = 0.07 two-sided)

That is suggestive but not conclusive at the usual 5% level. Had it been b = 30, c = 10, z = 20 / sqrt(40) = 3.16, a clear win. (This is the sign test / McNemar test with a normal approximation.)

Pitfalls

• Intervals computed from n items assume independent items; 50 near-duplicate prompts are not 50 samples.
• Running many comparisons and reporting the best one inflates false positives; pre-decide the comparison.
• Sampling noise is separate from run-to-run model noise at non-zero temperature: repeat the run a few times and report the spread as well.

Users submit a prompt, see two anonymous models’ responses side by side, and pick the better one (or tie). An Elo-style rating is fitted from these pairwise outcomes, the same idea used for chess rankings (arena-style leaderboards often fit a closely related Bradley-Terry model on all votes at once rather than updating one game at a time, but the interpretation of the rating gap is the same).

The formulas

E_A  = 1 / (1 + 10^((R_B - R_A) / 400))          expected score of A against B
R_A' = R_A + K * (S_A - E_A)                      rating after one game

Symbol	Meaning	Where it comes from
R_A, R_B	current ratings of A and B	start at a chosen value (e.g. 1500), then updated
E_A	A’s expected score: win probability, counting a tie as 0.5	derived from the rating gap
400	scale constant: a 400-point gap means the favourite wins about 91% of the time	chosen by convention (it is what makes 10^(gap/400) give nice odds)
S_A	actual result: 1 for a win, 0.5 tie, 0 loss	the human’s vote
K	step size: how far one result moves the rating	chosen (32 here; larger adapts faster but is noisier)

Worked example

def expected(ra, rb):
    return 1 / (1 + 10 ** ((rb - ra) / 400))

print(expected(1600, 1500))            # -> 0.640  (100-point favourite wins ~64%)
print(expected(1500, 1600))            # -> 0.360
print(expected(1700, 1500))            # -> 0.760
K = 32
print(1500 + K * (1 - expected(1500, 1500)))   # equal ratings, A wins  -> 1516.0
print(1600 + K * (1 - expected(1600, 1500)))   # favourite wins          -> 1611.5 (small gain)
print(1600 + K * (0 - expected(1600, 1500)))   # favourite LOSES         -> 1579.5 (big loss)

An upset moves ratings more than an expected result, which is what makes the system converge.

What it measures and what it misses

It measures aggregate human preference across a broad, organic, largely unfiltered distribution of real prompts, a genuinely valuable signal precisely because it is not a fixed, gameable question set. What it misses:

• Preference is not correctness: a confident, well-formatted, wrong answer often beats a hedged, correct one in a casual vote.
• Narrow expert capability: a domain specialist would judge very differently from a general crowd.
• It averages across wildly different task types into one number; a rating gap of 30 points is a small edge (the favourite wins about 54% of the time), not a big capability difference.
• It can be tuned for: length, formatting and tone that win casual comparisons (“arena-optimised” models).

Treat it as one hard-to-fake signal among several, not a substitute for task-specific evaluation.

1. Define “good” concretely. Not “is the answer good” but checkable properties: the extracted value is correct, the JSON validates, it cites a real source, it contains no banned claim.
2. Sample real inputs. Production logs if you have them; otherwise a deliberately diverse set covering common cases, edge cases and known failure modes, not only easy examples.
3. Choose the grader per property. Exact match or a programmatic check where possible (cheapest, most reliable); an LLM judge for open-ended quality (calibrated against human labels, see below); human review for the highest-stakes judgments.
4. Fix a baseline and re-run the same eval on every meaningful change (prompt, model, retrieval, parameters) so scores are comparable over time. This is regression testing, not a one-off report.
5. Size the set for the effect you care about (see Confidence intervals on accuracy). Detecting a 5-point improvement reliably needs hundreds of items; 20 only catches disasters.
6. Feed production failures back in. Every real bug becomes a new eval case.

A minimal, runnable eval harness

import json
from collections import defaultdict

cases = [
    {"id": 1, "cat": "invoice", "input": "Total due: EUR 1,250.00", "expect": "1250.00"},
    {"id": 2, "cat": "invoice", "input": "Amount: 99,5 EUR",        "expect": "99.50"},
    {"id": 3, "cat": "receipt", "input": "Paid 7.20 USD",           "expect": "7.20"},
]

def run_system(text: str) -> str:
    """Replace with your real model call; here a stub that fails on comma decimals."""
    import re
    m = re.search(r"(\d[\d,]*\.\d+)", text)
    return m.group(1).replace(",", "") if m else ""

per_cat = defaultdict(lambda: [0, 0])
failures = []
for c in cases:
    got = run_system(c["input"])
    ok = got == c["expect"]
    per_cat[c["cat"]][0] += ok
    per_cat[c["cat"]][1] += 1
    if not ok:
        failures.append({**c, "got": got})

for cat, (k, n) in per_cat.items():
    print(f"{cat}: {k}/{n}")
print(json.dumps(failures))
# invoice: 1/2
# receipt: 1/1
# [{"id": 2, "cat": "invoice", "input": "Amount: 99,5 EUR", "expect": "99.50", "got": ""}]

Walkthrough: each case has a category, so the report is per-category rather than one mean; failures are kept with full input and output so they can be inspected and committed to the set; the grader is a plain string comparison, which is the cheapest reliable choice when the answer is a single value. Save the output (and the git commit of prompt and model version) next to the date, so a later run can be compared against it.

Treating prompts as disposable inline strings (an f-string embedded directly wherever it’s used, with no history, no review process, no tests) means a “small wording tweak” someone makes casually can silently regress behavior on edge cases nobody thought to re-check, with no mechanism to notice until a user reports something odd. The more disciplined approach treats prompts closer to how code is treated: storing them as versioned templates in a defined location (not scattered inline), maintaining a regression test suite — a fixed, representative set of inputs with expected properties of the output (not necessarily an exact string match, since LLM output varies, but structural/behavioral assertions) — re-run automatically whenever a prompt changes, and subjecting prompt changes to the same code-review rigor as any other production change, precisely because prompt changes are just as capable of introducing regressions as code changes are, despite “just editing some text” feeling lower-stakes than it actually is.

Manual human review does not scale to production volumes of LLM output, and exact-match checks cannot grade open-ended quality, so teams automate the grading with another model. An LLM judge (often a separate, more capable model) reads a rubric plus one answer (pointwise) or two answers (pairwise) and returns a score or a winner. That makes evaluation feasible at a scale no human team could cover exhaustively, but the judge has its own blind spots and measured biases: it can be fooled by output that sounds confident and well structured even when it is factually wrong, and it systematically favours stylistic traits that do not track correctness.

Known biases and their mitigations

Bias	What happens	Mitigation
Positional	In pairwise judging, the answer in slot 1 (or 2) wins more often than chance	Judge both orderings; count a win only if it wins in both, otherwise tie
Length / verbosity	Longer, more elaborate answers score higher regardless of quality	Rubric says to ignore length; compare at matched length; include a length-controlled analysis
Self-preference	A judge rates outputs from its own model family higher	Use a judge from a different family than the candidates
Style over substance	Confident tone and tidy formatting beat hedged but accurate content	Provide a reference answer; require the judge to check specific facts
Rubric vagueness	“Rate quality 1-10” gives unstable, compressed scores	Binary or 3-level criteria with definitions and examples

Calibrating against humans

Score a sample (e.g. 100 items) with both the judge and real human raters, then measure agreement. Raw agreement misleads when one label dominates, so also compute Cohen’s kappa:

kappa = (p_o - p_e) / (1 - p_e)

Symbol	Meaning
p_o	observed agreement: the fraction of items where judge and human gave the same label
p_e	agreement expected by chance: P(judge good)*P(human good) + P(judge bad)*P(human bad), from each rater’s label frequencies
kappa	1 = perfect, 0 = no better than chance, negative = worse than chance

Worked example (illustrative): 100 items; both say good on 45, both say bad on 25, judge good but human bad on 20, judge bad but human good on 10.

p_o = (45 + 25) / 100 = 0.70
P(judge good) = 65/100 = 0.65     P(human good) = 55/100 = 0.55
p_e = 0.65*0.55 + 0.35*0.45 = 0.515
kappa = (0.70 - 0.515) / (1 - 0.515) = 0.38

70% raw agreement sounds fine, but kappa 0.38 is modest: the judge captures only about 38% of the agreement that was possible beyond chance. Inspect the 30 disagreements; here the judge is too lenient (20 judge-good/human-bad vs 10 the other way), so tighten the rubric, add reference answers, or switch to chain-of-thought scoring, then re-measure. Re-validate periodically against a fresh human-labelled sample, and always when you change the judge model, since behaviour drifts. Do not treat “the judge liked it” as ground truth for anything high-stakes without this step.

“Eyeballing” means running a handful of inputs, reading the outputs, and concluding “looks good”. It feels like testing but it is a very small, unrecorded, non-random sample judged by one person.

Failure mode 1: it misses minority regressions

If a change breaks a fraction r of inputs, the probability that n randomly chosen outputs contain none of the broken ones is (1 - r)^n. Each factor (1 - r) is the chance one output is fine; independence lets you multiply n of them.

Failure rate r	n = 5	n = 20	n = 50	n = 100
8%	66%	19%	1.5%	0.02%
2%	90%	67%	36%	13%

So with an 8% regression, five random outputs hide it 66% of the time. How many do you need to see it with 95% confidence? Solve (1 - r)^n = 0.05, i.e. n = ln(0.05) / ln(1 - r):

from math import log, ceil

def samples_to_catch(r: float, confidence: float = 0.95) -> int:
    """Items needed so that at least one failure is seen with the given confidence."""
    return ceil(log(1 - confidence) / log(1 - r))

for r in (0.20, 0.08, 0.02, 0.005):
    print(f"failure rate {r:5.1%}: need {samples_to_catch(r)} items")
# failure rate 20.0%: need 14 items
# failure rate  8.0%: need 36 items
# failure rate  2.0%: need 149 items
# failure rate  0.5%: need 598 items

Walkthrough: log(1 - confidence) is ln 0.05; log(1 - r) is the (negative) log of the per-item survival chance; the ratio, rounded up, is the sample size. Seeing a failure is not the same as measuring its rate: that needs the confidence-interval arithmetic above.

Failure mode 2: drift and no baseline

Nobody is measuring against a stable reference, so slow degradation is invisible. Example: you tweak the prompt five times, each “feels about the same”, and the cumulative effect is a 10-point loss on a category you stopped looking at.

Failure mode 3: selection and anchoring

You inspect the examples you remember or find convenient, which are the easy ones, and you read an output already expecting it to be fine. Also, reading is not grading: with no written criterion, the same output can feel acceptable today and wrong tomorrow.

Failure mode 4: no artifact

The next person, or you in three months, cannot reproduce the judgment or tell whether a later change improved things or merely felt different.

Worked contrast

Eyeballing:   5 inputs, read, "looks good"                 -> catches only failures >= ~45% frequent
Small eval:   60 saved inputs, 3 categories, checks coded  -> catches ~5% regressions, per category
Same eval re-run after every change, stored with git hash  -> trend over time

When it is genuinely fine

Throwaway prototyping where being wrong is cheap and iteration speed matters; exploring what the model can do at all; generating candidate failure cases to later turn into tests. The mistake is carrying the habit into anything shipped to real users.

The cheap upgrade path

The fix is rarely a huge rigorous suite up front. Save the handful of cases you already eyeballed as a permanent, re-runnable set (the harness above), write down what “pass” means for each in a line of code, add every future bug to it, and you have turned vibes into a signal you can track. Going from 5 to 50 cases is an afternoon of work and moves the failure rate you can reliably notice from about 45% down to about 6%.