Testing | kaguc — Writing to understand systems.

The testing pyramid for LLM applications: evals, not assertions

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 8 of the Engineering LLM Applications series. The evidence base is the same as the rest of the series: a production-grade AI writing agent (reads data → renders figures → generates LaTeX → compiles PDF), 351 test functions across the project.

The problem: non-deterministic output defeats assertions

The core move of conventional testing is asserting an exact value: assert f(x) == y. On an LLM main path this does not work — two generations from the same input are never verbatim-identical, so tests that assert fixed strings either stay red or get labeled flaky and lose all credibility. Teams commonly settle into one of two bad equilibria: test only the LLM-free edges and let the main path run naked, or force exact assertions and re-run until green. The real cost lands at change time: edit one word in a prompt and generation quality can silently degrade, with no signal in CI — part 11 lists this as failure mode four of a probabilistic component, “untestable regressions”.

The answer our project converged on fits in one sentence: an LLM application runs evals, not exact-value tests. Assertions are not abandoned — they change their object, from “the output equals X” to “the output satisfies property P”: structure score above a threshold, pairwise preference no worse than the previous version, a PDF that compiles, citations and numbers closed against their sources, metamorphic relations holding. Deterministic code keeps its exact assertions; the two kinds of test coexist in layers.

The philosophy: eval-driven development

Our methodology survey (docs/testing/01-research in the project, 2026-07, compiled from multi-source web search) compresses the field consensus into three points:

EDD (Eval-Driven Development): define the evaluation criteria before writing the agent; on every prompt/flow change → run the evals → read the score movement → decide. Evaluation is not post-hoc acceptance but the steering wheel of the development loop — TDD for agents.
The behavioral-testing triad (CheckList, ACL 2020): MFT (minimum functionality), INV (invariance), DIR (directional) tests. Transferred to this project: material with a planted gap must be flagged by the material check (MFT); deleting a data file must increase the gap count (DIR).
Metamorphic testing: when there is no reference answer, assert relations between inputs and outputs instead — “fuller material ⇒ structure score does not drop”, “delete half the data ⇒ more gap warnings”. This sidesteps the fundamental obstacle that non-deterministic output cannot be asserted exactly.

The tooling conclusion was no heavy dependencies: borrow DeepEval’s form — “an eval is a pytest case plus a threshold assertion” — and wrap our own scorers into pytest cases: zero new dependencies, offline-capable, self-hosted.

Four layers, split by “does it call a real LLM”

Layer	What runs	Calls LLM	Speed	How to run
L1 unit	Pure logic: parsing / indexing / routing / guards / sandbox whitelist / the scorers themselves / source verification	No	Seconds	`pytest`
L2 integration	Mock LLM drives the flow: full run_agent pipeline, tool loop, endpoint contracts	Mock	Seconds	`pytest`
smoke	Real local execution: matplotlib figure sandbox, xelatex compile	No	A few seconds	`pytest` (when xelatex is present)
L3 online eval	Real keys run the agent/judge; scorers plus thresholds decide	Yes (DeepSeek + Qwen)	Minutes	`pytest --run-live`

The layering criterion is not the textbook unit/integration split but two switches: does it call a real LLM, and does it need a real execution environment. A few design points:

How L2 is written: monkeypatch swaps agent._client for a fake, tool_calls and final content are scripted, and the test asserts flow events plus an eval_report score — it verifies “wired correctly”, not “high quality”. Note the scorer reuse: the same eval_report scores mock output in L2 and real output in L3, while being itself an L1 test subject — the scorer is tested before it is allowed to test anything else.
Tests never require compilation: pytest runs the Python source in seconds; the Nuitka build exists only to produce the customer-facing .exe. The dev loop = edit code → pytest → green.
Gating: @pytest.mark.live is skipped by default; only --run-live runs it, and missing keys auto-skip. The two real keys (DeepSeek for the text/agent main path, Aliyun Qwen for vision image reading) live in backend/.env, never in git. CI runs only the three offline layers by default; L3 runs manually, pre-release, or nightly.

pytest # L1+L2+smoke (default, seconds, offline, mandatory in CI)
pytest --run-live # add L3 online evals (real keys, slow, costs money)
python run_evals.py # one-shot golden-sample eval: scores vs threshold baseline

Objective criteria: five assertable properties

Agent output is never asserted against fixed strings; these properties are asserted instead:

Criterion	Implementation	Assertion form
Structure score	`eval_report.score_report`: 8 reference-free checks (compilable skeleton / ≥3 sections / error analysis / citation closure / data tables / equations / no AI boilerplate / no TODO), returns 0–1	score ≥ threshold
Pairwise preference	`eval_judge.judge_pairwise`: judged twice with positions swapped, ruled only if consistent	new version no worse than baseline
Compile pass	Real xelatex compile	PDF produced, page count > 0
Source closure	`verify.verify`: `\cite` closed against the bib, data numbers traceable to the material	no undefined citations; unsourced numbers bounded
Metamorphic relations	metamorphic / DIR	delete data → more gaps; fabricated number → more unsourced; fuller material → score does not drop

Two points deserve expansion. Pairwise preference must treat position bias: LLM judges systematically favor the first candidate (arXiv:2406.07791), so judge_pairwise evaluates twice with positions swapped and rules only on agreement; before judging anything, the judge must pass a reconstruction_accuracy self-check — correctly ranking known strong/weak pairs (≥ 0.7) — to earn the right to judge drafts. Validity engineering for the judge itself is the subject of part 9.

Metamorphic relations are the cheapest anti-hallucination criterion. The DIR test in the source-verification module, verbatim (string literals translated from the Chinese original):

def test_verify_metamorphic_fabricated_number_increases_unsourced():
 """DIR metamorphic: add a number absent from the material →
 the unsourced (suspected-fabricated) set must grow."""
 mat = "measured 1.0 and 2.0"
 base = verify.verify("values 1.0 and 2.0", mat)["numbers"]["unsourced"]
 more = verify.verify("values 1.0 and 2.0 and 7.77 from thin air", mat)["numbers"]["unsourced"]
 assert len(more) > len(base) and "7.77" in more

No knowledge of the “correct output” is needed — only a direction is asserted: one number appearing out of thin air must enlarge the unsourced set. The pattern replicates cheaply: delete a data file → more gap warnings; supply the .bib → the citation gap disappears.

The golden regression gate: the scorer must prove itself before it may gate

Online evals are the most faithful, but they need keys and take minutes — they cannot protect every change. So the regression gate is dual-track: beside the online gate run_evals.py (real generation plus judge), an offline gate test_golden_gate.py runs inside plain pytest, no keys needed, three gates:

Gate 1: scorer discriminative power. Gold-standard “good/bad” reports must be clearly separated by eval_report (assertion messages translated):

assert good["score"] >= 0.75, f"gold good report scored too low {good['score']}: {good['flags']}"
assert bad["score"] <= 0.35, f"gold bad report scored too high {bad['score']} (scorer lost discrimination)"
assert good["score"] - bad["score"] >= 0.4, "good/bad gap too small: scorer discrimination regressed"

The logical order matters: first prove the scorer actually encodes quality, only then allow it to act as a gate. A scorer whose discrimination has decayed renders every downstream threshold meaningless — and it will never raise an alarm about itself, which is why something else must pin it.

Gate 2: the gold-standard skeleton really compiles. The gold report skeleton must pass a real xelatex compile (skipped when no engine is installed locally; on the release machine / CI with TeX it is a hard gate). A generated report that fails to compile is the most direct signal of prompt or template regression; this gate protects the template plus the compile chain.

Gate 3: prompt shape. Offline tests cannot run a real LLM and cannot judge output quality — but they can judge the structure of a prompt: the scoped prompt build_user_scoped must still demand “rewrite only this section”, with \documentclass and \end{document} appearing inside the “strictly forbidden to output” clause; the full-document path build_user(gaps=None) must be byte-identical to the call without gaps. The moment someone edits the prompt back to “output the whole document”, plain pytest goes red.

(One additional foundation check: the gold-standard report must parse correctly under the deterministic sectionizer — protecting the base of section-level editing.)

The baseline: every score pinned to git_sha and prompt_hash

The online gate run_evals.py outputs more than red/green: every real run appends its scores to docs/testing/eval_baseline.tsv, and its exit code feeds CI. The file’s actual contents (fixture names translated):

date	git_sha	prompt_hash	metric	name	score	threshold	pass
2026-07-01 21:49	cb5bba9	6f3202abff3a	judge.recon	strong/weak pair	1.000	0.70	1
2026-07-03 11:29	39b9e29	46bed3f78ca1	report.score	field-sweep S-params	0.875	0.50	1
2026-07-03 11:29	39b9e29	46bed3f78ca1	report.compiles	field-sweep S-params	1.000	1.00	1
2026-07-03 11:29	39b9e29	46bed3f78ca1	judge.recon	strong/weak pair	1.000	0.70	1

Every row pins both git_sha and prompt_hash: when a score drifts, the first question — “did the code change or did the prompt change” — is answerable immediately. The full development loop:

flowchart TD
 A[Edit code / edit prompt] --> B{pytest: L1+L2+smoke, seconds}
 B -->|red| A
 B -->|green, quality-relevant code untouched| Z[Commit]
 B -->|green, agent/quality-relevant code touched| C[pytest --run-live or run_evals.py]
 C --> D[Real agent run → scorer/judge scores]
 D --> E{Compare against thresholds and eval_baseline.tsv}
 E -->|meets thresholds| F[Green: scores appended to baseline] --> Z
 E -->|regression| A
 Z --> R[Pre-release: default suite + --run-live all green → only then the Nuitka build]

Measured numbers from the rollout: on the day the system landed (2026-07-01), the default suite ran 144 passed + 7 skipped (live) and --run-live passed 7/7; two days later the golden gate was added, 248 passed with no regressions; at the time of writing the project holds 351 test functions. Flakiness governance comes down to two moves: LLM cases assert thresholds/intervals/metamorphic relations rather than exact strings, and the judge uses swap-averaging to reduce variance.

Where this does not apply

A small golden set is not a proof of quality. The survey notes that aggregate metrics need on the order of hundreds of examples to be trustworthy (that figure not independently verified); this project’s online golden set started with 1 report fixture plus 2 strong/weak pairs. At that scale the online eval is a smoke gate — it catches large regressions — not a quality metric. Do not conclude “quality improved” from it before the set grows.
Objective criteria only measure the measurable. The 8-item structure score is a reference-free floor: it judges “is this report solid and compilable”, not “is it well written”. Preference-level quality above the floor needs a judge — and the judge must be validity-tested first; an unvalidated judge gate is more dangerous than no gate.
Green L2 is not green quality. Mock-driven integration tests verify wiring and flow events; they know nothing about the quality of real model output. Reading L2 passes as a quality signal is the most common misreading of this layering.
Online evals are slow, cost money, and carry variance — keep them out of the fast loop. They belong pre-release, nightly, or after quality-relevant changes. And if the product is still in prototype phase with prompts rewritten daily, the threshold-maintenance cost of golden gates will exceed their value — establish a reasonably stable product definition first, then build the gates.

References

CheckList (ACL 2020) — the MFT/INV/DIR behavioral-testing triad; the direct source of this article’s criteria design.
arXiv:2406.07791 — systematic study of position bias in LLM judges (systematic preference for the first candidate).
arXiv:2410.15393 — judge calibration methods: swap-and-average, balanced-position calibration, and others.
arXiv:2504.18827 — metamorphic testing applied to LLMs (LLMorph and related work).
DeepEval — the “eval as a pytest case + threshold assertion” form we borrowed (without taking the dependency).

These external sources come from the project’s research document (2026-07, compiled from multi-source web search); apart from this project’s own measurements, second-hand claims were not independently re-verified.

Validity engineering for LLM-as-judge

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 9 of the Engineering LLM Applications series. The first-hand data comes from a writing-judgment distillation experiment (whose design and headline results are the subject of Part 12); every number is taken from that project’s experiment journal, run ledger, and source code.

The problem: the judge is itself an unmeasured probabilistic component

Pairwise judging (pairwise LLM-as-judge) is the mainstream way to evaluate LLM application quality: show the model drafts A and B, ask which is better, tally win rates. It sidesteps the calibration problems of absolute scoring — and introduces a subtler one: the judge is a probabilistic component too. Why should its verdicts be trusted?

This is not a hypothetical risk. It happened twice in our experiment. The primary judge of the first round was llama-3.3-70b; measured after the fact, its QWK against the quality anchor was 0.10 — near random — and the entire round was voided. In a later round, claude-sonnet-4.6 ruled B>A while gpt-4o ruled A>B on the same drafts; the aggregate verdict could only be recorded as “ambiguous”. If you cannot answer “which judge is credible”, the win-rate table is just another array of random numbers.

This article names the practice we converged on validity engineering: before consuming any verdict from a judge, establish with independent evidence that the judge measures what you think it measures. It consists of five mechanisms: admission, anchor independence, scale diagnosis, bias control, and a pre-deployment acceptance gate.

Mechanism 1: judge admission — human-score anchors and a QWK bar

Admission works like this: have each candidate judge score a set of anchor texts that carry quality scores, then compute QWK (Quadratic Weighted Kappa, an ordinal-agreement metric) against the anchor scores. The bar is written directly into the module docstring of qwk_anchor.py: “≥0.5 → the judge is credible and pairwise win-rate conclusions are worth believing; <0.3 → go back and fix the rubric first”, with 0.3–0.5 recorded as “marginal”.

Measured results for the four candidate judges:

Judge	QWK	Ruling
claude-sonnet-4.6	0.55–0.61	Retained (primary judge)
qwen3.7-max	0.571	Retained (cross judge)
gpt-4o	0.281	Eliminated (<0.3, not credible)
llama-3.3-70b	0.10	Eliminated; the first round it served as primary judge was voided in full

Elimination is not academic fastidiousness: gpt-4o was one party in the “two judges, opposite verdicts” incident above, and removing it returned the ruling to the discriminative power of the credible judges. But this round of QWK was computed against a synthetic anchor — which leads to the next mechanism, and the most expensive lesson in the whole system.

Mechanism 2: the anchor must be independent of the system under evaluation — a circular-validation incident

The incident was triggered by one user question: “is the test data even representative?” The audit found that none of the three real datasets the plan specified (arXivEdits / peS2o / ASAP) had ever been downloaded successfully — a hardcoded URL had long 404’d, the Kaggle login step was never done, and the download script ignored the system proxy. Everything had run on seed and synthetic fallback data: 13 exemplars, 5 hand-written gold pairs, 8 synthetic pairs, and a 39-item anchor whose scores were synthetic, not human.

The consequence was structural: the so-called “QWK anchor” was in fact the judge aligning with the program’s own degradation rules — circular self-validation. Historical QWK of 0.51–0.61 was voided in full; it constitutes no human-grounded credibility whatsoever. One asymmetry is worth stating: what the incident voided were the “pass” rulings; the “out” rulings actually stand — a judge that cannot even align with the coarse ordering of a synthetic anchor is even less likely to align with humans (our inference, not an independent experiment).

The fix was to actually connect the real data: 200 genuine expert-revision pairs from arXivEdits (filtered by the intention field for writing-quality edits, discarding trivial typo fixes; median length difference 27 characters), and 120 essays with real human holistic scores from ASAP 2.0 (stratified sampling, 20 per score band 1–6, guaranteeing QWK has range to work with). The lesson in one sentence: the anchor must come from outside the system under evaluation, or QWK measures the judge’s agreement with your own code.

Mechanism 3: joint QWK×Spearman diagnosis — scale compression is not ranking failure

With the human anchor in place, the first real QWK came out at 0.323: model mean 2.32 vs human mean 3.50, a systematic severity bias of about 1.2 points. A multi-agent investigation diagnosed three root causes: severity-biased scale misalignment; a genre/rubric mismatch (a physics-report rubric grading English argumentative essays); and the inherently low ceiling of single-essay absolute scoring as a usage mode (the project’s investigation cited roughly 0.60 as the zero-shot ceiling for ASAP holistic scoring — a reference value, not independently verified).

The fix took the zero/low-cost route: switch to the ASAP holistic six-band rubric, delete all “strict” wording (the system prompt changed from “strict” to “well-calibrated”), add a full-range prior; few-shot examples stayed disabled behind a comment, because the anchor data had no train/eval split and would leak. Before and after:

Metric	Before	After
QWK	0.323	0.456
Spearman	—	0.813
Pearson	—	0.788
Systematic bias	-1.18	-0.87
Model mean	2.32	2.63 (human 3.50)

QWK still did not clear 0.5 — but the key finding sits in another number: Spearman 0.813 ≫ QWK 0.456. The diagnostic rule in qwk_anchor.py summarizes it in one line: “high Spearman with low QWK → a pure scale problem (recalibration can save it); both low → the ranking signal itself is weak.” The confusion matrix supplied the hard evidence — the judge never awards a 5 or a 6; all 40 essays humans scored 5 or 6 were compressed into the 3–4 columns. This is a strong-ranking, conservative-scoring judge, not one scoring at random.

The conclusion lands on usage: pairwise judging consumes only the ranking (the judge’s strength), never the absolute scale (its weakness) — so pairwise win rates stand on solid ground as the primary signal, while single-essay absolute scores are unusable. The scale can be partially corrected by post-hoc monotonic recalibration, but its legitimacy has three preconditions (confirmed by adversarial verification): the mapping must be monotone, numbers must be reported on held-out/CV data, and parameters must not be transferred across domains — a 5-fold reproduction showed in-sample recalibration inflates the number and held-out evaluation automatically strips the inflation away. And recalibration cannot rescue the top of the scale: humans’ 5s and 6s are completely indistinguishable to the judge (both get 3s and 4s) — a genuine ceiling.

Mechanism 4: bias control — position swapping, identity masking, a rubric that penalizes verbosity

LLM judges have position preferences (favoring whichever draft appears first or last). The countermeasure is protocol-level: every pair is judged twice, in the orders (A,B) and (B,A), and if the two rulings disagree, the pair is recorded as a tie — never a fake verdict. The core logic is four lines (eval_judge.py):

sys_ab = _MAP_AB[_one_order(domain, a, b, rubric, cf)] # order 1: first=A
sys_ba = _MAP_BA[_one_order(domain, b, a, rubric, cf)] # order 2: first=B
consistent = (sys_ab == sys_ba)
decided = sys_ab if (consistent and sys_ab != "tie") else "tie"

The design yields a judge-quality signal for free: swap consistency. Measured, claude-sonnet-4.6 was swap-consistent on 93% of pairs at N=30 (28/30), and 8/8 in the N=8 pilot; qwen3.7-max mostly ranged 0.2–0.5, its win rates dominated by ties — the protocol automatically absorbs a weak judge’s instability into ties instead of letting it contaminate conclusions.

Two companion measures: identity masking (the judge cannot see which pipeline produced which draft), and length bias treated by rubric rather than by hope — the judging prompt states explicitly that “longer is not a virtue; low information density is penalized as verbosity; when substantively equal, the longer one must not win for being longer.”

Mechanism 5: reconstruction acceptance — judge known strong/weak pairs before judging new drafts

Beyond admission and diagnosis there is one final gate before deployment, called “ICAI-style acceptance” (reconstruction) inside the project: hand the judge known strong/weak pairs (gold-standard good drafts vs deliberately degraded weak ones) along with its rubric and have it pick “the better one”; only a pick-correct rate ≥ 0.7 (the code constant GATE_THRESHOLD = 0.7) qualifies it to judge new drafts, with swap consistency reported alongside. The logic is plain: a judge that cannot pick correctly when the answer is known has no standing to rule when it is not.

The cost boundary is also written into the module docstring: two swap calls per pair are real LLM calls and burn tokens — so this gate lives on the development/offline side and does not run in the client’s real-time path by default.

The whole picture: a judge’s path to deployment

flowchart TD
 A[Candidate judge] --> B[Score human-anchor set → QWK]
 B -->|QWK < 0.3| X[Eliminated]
 B -->|clears the bar| C{Joint QWK × Spearman diagnosis}
 C -->|ranking weak too| Y[Fix rubric / few-shot, re-test]
 C -->|ranking strong, scale compressed| D[License pairwise judging only<br/>absolute scoring unusable]
 D --> E[Reconstruction gate: known-pair pick-correct rate ≥ 0.7]
 E -->|fail| Y
 E -->|pass| F[Deploy: swap-order double judging<br/>disagreement = tie, identity masked, rubric penalizes verbosity]

Where this does not apply

In keeping with this series’ convention — the limits of the practice, and what it does not solve:

Same-family judge self-preference must be declared. Our primary judge, claude-sonnet-4.6, belongs to the same family as the Claude Opus models that performed skill distillation and adversarial auditing; every “strong” conclusion of the experiment comes from this same-family judge, carrying a “Claude recognizes Claude-style reasoning” self-preference risk. There are only two ways out: a third-party judge from a different family, or human-anchor re-verification.
The anchor’s domain is not the target domain. An English anchor only certifies the judge on English argumentative essays; it certifies nothing about judging Chinese physics reports — an external-validity gap. The production-side port hard-codes this warning in its docstring: the pairwise judge’s reliability in the Chinese domain is unverified, and it does not run in the real-time path by default.
High-band discrimination is unverified. Spearman 0.81 is driven mainly by the mid-to-low score range; yet pairwise comparison happens precisely in the “neither draft is bad” region — the judge’s ranking ability there is an untested assumption, and the upper-range collapse is a real threat to it.
An LLM judge does not replace a human anchor. When all preference evidence comes from LLM judges, systematic LLM-judge preferences cannot be ruled out; the experiment’s human blind evaluation (30 pairs) was still being collected at the time of writing, and is honestly listed as the most critical limitation.
Do not use a judge for deterministically detectable failures. Content loss and structural degradation are handled by a zero-token regression gate (see Part 11) — more correct, and one judge call cheaper per round. The judge belongs only at the preference layer, where deterministic criteria cannot reach.

References

arXivEdits — a dataset of genuine expert revisions (schema: sentence-1/sentence-2 + intention); the project filtered by intention for writing-quality edits, taking 200 pairs as strong/weak pairs.
ASAP-AES 2.0 — the English essay benchmark with real human holistic scores (1–6); obtained via the Hugging Face mirror jatinmehra/Automated-Essay-Scoring-2.0, with 120 essays stratified-sampled as the credibility anchor.
First-hand records: every number in this article comes from the project’s experiment journal and machine-readable run ledger (journal / runs.tsv); per-essay (id, human, model) scores are persisted to results/qwk_scores.csv to support recomputation and recalibration.