Validity engineering for LLM-as-judge

Also in 中文. Part 9 of the Engineering LLM Applications series. The first-hand data comes from a writing-judgment distillation experiment (whose design and headline results are the subject of Part 12); every number is taken from that project’s experiment journal, run ledger, and source code.

The problem: the judge is itself an unmeasured probabilistic component

Pairwise judging (pairwise LLM-as-judge) is the mainstream way to evaluate LLM application quality: show the model drafts A and B, ask which is better, tally win rates. It sidesteps the calibration problems of absolute scoring — and introduces a subtler one: the judge is a probabilistic component too. Why should its verdicts be trusted?

This is not a hypothetical risk. It happened twice in our experiment. The primary judge of the first round was llama-3.3-70b; measured after the fact, its QWK against the quality anchor was 0.10 — near random — and the entire round was voided. In a later round, claude-sonnet-4.6 ruled B>A while gpt-4o ruled A>B on the same drafts; the aggregate verdict could only be recorded as “ambiguous”. If you cannot answer “which judge is credible”, the win-rate table is just another array of random numbers.

This article names the practice we converged on validity engineering: before consuming any verdict from a judge, establish with independent evidence that the judge measures what you think it measures. It consists of five mechanisms: admission, anchor independence, scale diagnosis, bias control, and a pre-deployment acceptance gate.

Mechanism 1: judge admission — human-score anchors and a QWK bar

Admission works like this: have each candidate judge score a set of anchor texts that carry quality scores, then compute QWK (Quadratic Weighted Kappa, an ordinal-agreement metric) against the anchor scores. The bar is written directly into the module docstring of qwk_anchor.py: “≥0.5 → the judge is credible and pairwise win-rate conclusions are worth believing; <0.3 → go back and fix the rubric first”, with 0.3–0.5 recorded as “marginal”.

Measured results for the four candidate judges:

Elimination is not academic fastidiousness: gpt-4o was one party in the “two judges, opposite verdicts” incident above, and removing it returned the ruling to the discriminative power of the credible judges. But this round of QWK was computed against a synthetic anchor — which leads to the next mechanism, and the most expensive lesson in the whole system.

Mechanism 2: the anchor must be independent of the system under evaluation — a circular-validation incident

The incident was triggered by one user question: “is the test data even representative?” The audit found that none of the three real datasets the plan specified (arXivEdits / peS2o / ASAP) had ever been downloaded successfully — a hardcoded URL had long 404’d, the Kaggle login step was never done, and the download script ignored the system proxy. Everything had run on seed and synthetic fallback data: 13 exemplars, 5 hand-written gold pairs, 8 synthetic pairs, and a 39-item anchor whose scores were synthetic, not human.

The consequence was structural: the so-called “QWK anchor” was in fact the judge aligning with the program’s own degradation rules — circular self-validation. Historical QWK of 0.51–0.61 was voided in full; it constitutes no human-grounded credibility whatsoever. One asymmetry is worth stating: what the incident voided were the “pass” rulings; the “out” rulings actually stand — a judge that cannot even align with the coarse ordering of a synthetic anchor is even less likely to align with humans (our inference, not an independent experiment).

The fix was to actually connect the real data: 200 genuine expert-revision pairs from arXivEdits (filtered by the intention field for writing-quality edits, discarding trivial typo fixes; median length difference 27 characters), and 120 essays with real human holistic scores from ASAP 2.0 (stratified sampling, 20 per score band 1–6, guaranteeing QWK has range to work with). The lesson in one sentence: the anchor must come from outside the system under evaluation, or QWK measures the judge’s agreement with your own code.

Mechanism 3: joint QWK×Spearman diagnosis — scale compression is not ranking failure

With the human anchor in place, the first real QWK came out at 0.323: model mean 2.32 vs human mean 3.50, a systematic severity bias of about 1.2 points. A multi-agent investigation diagnosed three root causes: severity-biased scale misalignment; a genre/rubric mismatch (a physics-report rubric grading English argumentative essays); and the inherently low ceiling of single-essay absolute scoring as a usage mode (the project’s investigation cited roughly 0.60 as the zero-shot ceiling for ASAP holistic scoring — a reference value, not independently verified).

The fix took the zero/low-cost route: switch to the ASAP holistic six-band rubric, delete all “strict” wording (the system prompt changed from “strict” to “well-calibrated”), add a full-range prior; few-shot examples stayed disabled behind a comment, because the anchor data had no train/eval split and would leak. Before and after:

QWK still did not clear 0.5 — but the key finding sits in another number: Spearman 0.813 ≫ QWK 0.456. The diagnostic rule in qwk_anchor.py summarizes it in one line: “high Spearman with low QWK → a pure scale problem (recalibration can save it); both low → the ranking signal itself is weak.” The confusion matrix supplied the hard evidence — the judge never awards a 5 or a 6; all 40 essays humans scored 5 or 6 were compressed into the 3–4 columns. This is a strong-ranking, conservative-scoring judge, not one scoring at random.

The conclusion lands on usage: pairwise judging consumes only the ranking (the judge’s strength), never the absolute scale (its weakness) — so pairwise win rates stand on solid ground as the primary signal, while single-essay absolute scores are unusable. The scale can be partially corrected by post-hoc monotonic recalibration, but its legitimacy has three preconditions (confirmed by adversarial verification): the mapping must be monotone, numbers must be reported on held-out/CV data, and parameters must not be transferred across domains — a 5-fold reproduction showed in-sample recalibration inflates the number and held-out evaluation automatically strips the inflation away. And recalibration cannot rescue the top of the scale: humans’ 5s and 6s are completely indistinguishable to the judge (both get 3s and 4s) — a genuine ceiling.

Mechanism 4: bias control — position swapping, identity masking, a rubric that penalizes verbosity

LLM judges have position preferences (favoring whichever draft appears first or last). The countermeasure is protocol-level: every pair is judged twice, in the orders (A,B) and (B,A), and if the two rulings disagree, the pair is recorded as a tie — never a fake verdict. The core logic is four lines (eval_judge.py):

sys_ab = _MAP_AB[_one_order(domain, a, b, rubric, cf)]   # order 1: first=A
sys_ba = _MAP_BA[_one_order(domain, b, a, rubric, cf)]   # order 2: first=B
consistent = (sys_ab == sys_ba)
decided = sys_ab if (consistent and sys_ab != "tie") else "tie"

The design yields a judge-quality signal for free: swap consistency. Measured, claude-sonnet-4.6 was swap-consistent on 93% of pairs at N=30 (28/30), and 8/8 in the N=8 pilot; qwen3.7-max mostly ranged 0.2–0.5, its win rates dominated by ties — the protocol automatically absorbs a weak judge’s instability into ties instead of letting it contaminate conclusions.

Two companion measures: identity masking (the judge cannot see which pipeline produced which draft), and length bias treated by rubric rather than by hope — the judging prompt states explicitly that “longer is not a virtue; low information density is penalized as verbosity; when substantively equal, the longer one must not win for being longer.”

Mechanism 5: reconstruction acceptance — judge known strong/weak pairs before judging new drafts

Beyond admission and diagnosis there is one final gate before deployment, called “ICAI-style acceptance” (reconstruction) inside the project: hand the judge known strong/weak pairs (gold-standard good drafts vs deliberately degraded weak ones) along with its rubric and have it pick “the better one”; only a pick-correct rate ≥ 0.7 (the code constant GATE_THRESHOLD = 0.7) qualifies it to judge new drafts, with swap consistency reported alongside. The logic is plain: a judge that cannot pick correctly when the answer is known has no standing to rule when it is not.

The cost boundary is also written into the module docstring: two swap calls per pair are real LLM calls and burn tokens — so this gate lives on the development/offline side and does not run in the client’s real-time path by default.

The whole picture: a judge’s path to deployment

Where this does not apply

In keeping with this series’ convention — the limits of the practice, and what it does not solve:

1. Same-family judge self-preference must be declared. Our primary judge, claude-sonnet-4.6, belongs to the same family as the Claude Opus models that performed skill distillation and adversarial auditing; every “strong” conclusion of the experiment comes from this same-family judge, carrying a “Claude recognizes Claude-style reasoning” self-preference risk. There are only two ways out: a third-party judge from a different family, or human-anchor re-verification.
2. The anchor’s domain is not the target domain. An English anchor only certifies the judge on English argumentative essays; it certifies nothing about judging Chinese physics reports — an external-validity gap. The production-side port hard-codes this warning in its docstring: the pairwise judge’s reliability in the Chinese domain is unverified, and it does not run in the real-time path by default.
3. High-band discrimination is unverified. Spearman 0.81 is driven mainly by the mid-to-low score range; yet pairwise comparison happens precisely in the “neither draft is bad” region — the judge’s ranking ability there is an untested assumption, and the upper-range collapse is a real threat to it.
4. An LLM judge does not replace a human anchor. When all preference evidence comes from LLM judges, systematic LLM-judge preferences cannot be ruled out; the experiment’s human blind evaluation (30 pairs) was still being collected at the time of writing, and is honestly listed as the most critical limitation.
5. Do not use a judge for deterministically detectable failures. Content loss and structural degradation are handled by a zero-token regression gate (see Part 11) — more correct, and one judge call cheaper per round. The judge belongs only at the preference layer, where deterministic criteria cannot reach.

References

• arXivEdits — a dataset of genuine expert revisions (schema: sentence-1/sentence-2 + intention); the project filtered by intention for writing-quality edits, taking 200 pairs as strong/weak pairs.
• ASAP-AES 2.0 — the English essay benchmark with real human holistic scores (1–6); obtained via the Hugging Face mirror jatinmehra/Automated-Essay-Scoring-2.0, with 120 essays stratified-sampled as the credibility anchor.
• First-hand records: every number in this article comes from the project’s experiment journal and machine-readable run ledger (journal / runs.tsv); per-essay (id, human, model) scores are persisted to results/qwk_scores.csv to support recomputation and recalibration.