The testing pyramid for LLM applications: evals, not assertions

Also in 中文. Part 8 of the Engineering LLM Applications series. The evidence base is the same as the rest of the series: a production-grade AI writing agent (reads data → renders figures → generates LaTeX → compiles PDF), 351 test functions across the project.

The problem: non-deterministic output defeats assertions

The core move of conventional testing is asserting an exact value: assert f(x) == y. On an LLM main path this does not work — two generations from the same input are never verbatim-identical, so tests that assert fixed strings either stay red or get labeled flaky and lose all credibility. Teams commonly settle into one of two bad equilibria: test only the LLM-free edges and let the main path run naked, or force exact assertions and re-run until green. The real cost lands at change time: edit one word in a prompt and generation quality can silently degrade, with no signal in CI — part 11 lists this as failure mode four of a probabilistic component, “untestable regressions”.

The answer our project converged on fits in one sentence: an LLM application runs evals, not exact-value tests. Assertions are not abandoned — they change their object, from “the output equals X” to “the output satisfies property P”: structure score above a threshold, pairwise preference no worse than the previous version, a PDF that compiles, citations and numbers closed against their sources, metamorphic relations holding. Deterministic code keeps its exact assertions; the two kinds of test coexist in layers.

The philosophy: eval-driven development

Our methodology survey (docs/testing/01-research in the project, 2026-07, compiled from multi-source web search) compresses the field consensus into three points:

1. EDD (Eval-Driven Development): define the evaluation criteria before writing the agent; on every prompt/flow change → run the evals → read the score movement → decide. Evaluation is not post-hoc acceptance but the steering wheel of the development loop — TDD for agents.
2. The behavioral-testing triad (CheckList, ACL 2020): MFT (minimum functionality), INV (invariance), DIR (directional) tests. Transferred to this project: material with a planted gap must be flagged by the material check (MFT); deleting a data file must increase the gap count (DIR).
3. Metamorphic testing: when there is no reference answer, assert relations between inputs and outputs instead — “fuller material ⇒ structure score does not drop”, “delete half the data ⇒ more gap warnings”. This sidesteps the fundamental obstacle that non-deterministic output cannot be asserted exactly.

The tooling conclusion was no heavy dependencies: borrow DeepEval’s form — “an eval is a pytest case plus a threshold assertion” — and wrap our own scorers into pytest cases: zero new dependencies, offline-capable, self-hosted.

Four layers, split by “does it call a real LLM”

The layering criterion is not the textbook unit/integration split but two switches: does it call a real LLM, and does it need a real execution environment. A few design points:

• How L2 is written: monkeypatch swaps agent._client for a fake, tool_calls and final content are scripted, and the test asserts flow events plus an eval_report score — it verifies “wired correctly”, not “high quality”. Note the scorer reuse: the same eval_report scores mock output in L2 and real output in L3, while being itself an L1 test subject — the scorer is tested before it is allowed to test anything else.
• Tests never require compilation: pytest runs the Python source in seconds; the Nuitka build exists only to produce the customer-facing .exe. The dev loop = edit code → pytest → green.
• Gating: @pytest.mark.live is skipped by default; only --run-live runs it, and missing keys auto-skip. The two real keys (DeepSeek for the text/agent main path, Aliyun Qwen for vision image reading) live in backend/.env, never in git. CI runs only the three offline layers by default; L3 runs manually, pre-release, or nightly.

pytest                # L1+L2+smoke (default, seconds, offline, mandatory in CI)
pytest --run-live     # add L3 online evals (real keys, slow, costs money)
python run_evals.py   # one-shot golden-sample eval: scores vs threshold baseline

Objective criteria: five assertable properties

Agent output is never asserted against fixed strings; these properties are asserted instead:

Two points deserve expansion. Pairwise preference must treat position bias: LLM judges systematically favor the first candidate (arXiv:2406.07791), so judge_pairwise evaluates twice with positions swapped and rules only on agreement; before judging anything, the judge must pass a reconstruction_accuracy self-check — correctly ranking known strong/weak pairs (≥ 0.7) — to earn the right to judge drafts. Validity engineering for the judge itself is the subject of part 9.

Metamorphic relations are the cheapest anti-hallucination criterion. The DIR test in the source-verification module, verbatim (string literals translated from the Chinese original):

def test_verify_metamorphic_fabricated_number_increases_unsourced():
    """DIR metamorphic: add a number absent from the material →
    the unsourced (suspected-fabricated) set must grow."""
    mat = "measured 1.0 and 2.0"
    base = verify.verify("values 1.0 and 2.0", mat)["numbers"]["unsourced"]
    more = verify.verify("values 1.0 and 2.0 and 7.77 from thin air", mat)["numbers"]["unsourced"]
    assert len(more) > len(base) and "7.77" in more

No knowledge of the “correct output” is needed — only a direction is asserted: one number appearing out of thin air must enlarge the unsourced set. The pattern replicates cheaply: delete a data file → more gap warnings; supply the .bib → the citation gap disappears.

The golden regression gate: the scorer must prove itself before it may gate

Online evals are the most faithful, but they need keys and take minutes — they cannot protect every change. So the regression gate is dual-track: beside the online gate run_evals.py (real generation plus judge), an offline gate test_golden_gate.py runs inside plain pytest, no keys needed, three gates:

Gate 1: scorer discriminative power. Gold-standard “good/bad” reports must be clearly separated by eval_report (assertion messages translated):

assert good["score"] >= 0.75, f"gold good report scored too low {good['score']}: {good['flags']}"
assert bad["score"] <= 0.35, f"gold bad report scored too high {bad['score']} (scorer lost discrimination)"
assert good["score"] - bad["score"] >= 0.4, "good/bad gap too small: scorer discrimination regressed"

The logical order matters: first prove the scorer actually encodes quality, only then allow it to act as a gate. A scorer whose discrimination has decayed renders every downstream threshold meaningless — and it will never raise an alarm about itself, which is why something else must pin it.

Gate 2: the gold-standard skeleton really compiles. The gold report skeleton must pass a real xelatex compile (skipped when no engine is installed locally; on the release machine / CI with TeX it is a hard gate). A generated report that fails to compile is the most direct signal of prompt or template regression; this gate protects the template plus the compile chain.

Gate 3: prompt shape. Offline tests cannot run a real LLM and cannot judge output quality — but they can judge the structure of a prompt: the scoped prompt build_user_scoped must still demand “rewrite only this section”, with \documentclass and \end{document} appearing inside the “strictly forbidden to output” clause; the full-document path build_user(gaps=None) must be byte-identical to the call without gaps. The moment someone edits the prompt back to “output the whole document”, plain pytest goes red.

(One additional foundation check: the gold-standard report must parse correctly under the deterministic sectionizer — protecting the base of section-level editing.)

The baseline: every score pinned to git_sha and prompt_hash

The online gate run_evals.py outputs more than red/green: every real run appends its scores to docs/testing/eval_baseline.tsv, and its exit code feeds CI. The file’s actual contents (fixture names translated):

Every row pins both git_sha and prompt_hash: when a score drifts, the first question — “did the code change or did the prompt change” — is answerable immediately. The full development loop:

Measured numbers from the rollout: on the day the system landed (2026-07-01), the default suite ran 144 passed + 7 skipped (live) and --run-live passed 7/7; two days later the golden gate was added, 248 passed with no regressions; at the time of writing the project holds 351 test functions. Flakiness governance comes down to two moves: LLM cases assert thresholds/intervals/metamorphic relations rather than exact strings, and the judge uses swap-averaging to reduce variance.

Where this does not apply

1. A small golden set is not a proof of quality. The survey notes that aggregate metrics need on the order of hundreds of examples to be trustworthy (that figure not independently verified); this project’s online golden set started with 1 report fixture plus 2 strong/weak pairs. At that scale the online eval is a smoke gate — it catches large regressions — not a quality metric. Do not conclude “quality improved” from it before the set grows.
2. Objective criteria only measure the measurable. The 8-item structure score is a reference-free floor: it judges “is this report solid and compilable”, not “is it well written”. Preference-level quality above the floor needs a judge — and the judge must be validity-tested first; an unvalidated judge gate is more dangerous than no gate.
3. Green L2 is not green quality. Mock-driven integration tests verify wiring and flow events; they know nothing about the quality of real model output. Reading L2 passes as a quality signal is the most common misreading of this layering.
4. Online evals are slow, cost money, and carry variance — keep them out of the fast loop. They belong pre-release, nightly, or after quality-relevant changes. And if the product is still in prototype phase with prompts rewritten daily, the threshold-maintenance cost of golden gates will exceed their value — establish a reasonably stable product definition first, then build the gates.

References

• CheckList (ACL 2020) — the MFT/INV/DIR behavioral-testing triad; the direct source of this article’s criteria design.
• arXiv:2406.07791 — systematic study of position bias in LLM judges (systematic preference for the first candidate).
• arXiv:2410.15393 — judge calibration methods: swap-and-average, balanced-position calibration, and others.
• arXiv:2504.18827 — metamorphic testing applied to LLMs (LLMorph and related work).
• DeepEval — the “eval as a pytest case + threshold assertion” form we borrowed (without taking the dependency).

These external sources come from the project’s research document (2026-07, compiled from multi-source web search); apart from this project’s own measurements, second-hand claims were not independently re-verified.