# The testing pyramid for LLM applications: evals, not assertions

> Non-deterministic output defeats exact-value assertions. This article presents a four-layer testing pyramid — zero-LLM unit tests, mock-driven integration, real-execution smoke, and --run-live gated online evals — plus five objective criteria (structure score, pairwise preference, compile pass, source closure, metamorphic relations), a golden regression gate that makes the scorer prove its own discriminative power first, and a score baseline pinned to git_sha + prompt_hash. The evidence base is a production-grade writing agent with 351 tests.

- Canonical (HTML): https://kaguc.com/blog/llm-testing-pyramid/
- Date: 2026-07-29


*Also in [中文](/blog/llm-testing-pyramid-zh/). Part 8 of the Engineering LLM Applications series. The evidence base is the same as the rest of the series: a production-grade AI writing agent (reads data → renders figures → generates LaTeX → compiles PDF), 351 test functions across the project.*

## The problem: non-deterministic output defeats assertions

The core move of conventional testing is asserting an exact value: `assert f(x) == y`. On an LLM main path this does not work — two generations from the same input are never verbatim-identical, so tests that assert fixed strings either stay red or get labeled flaky and lose all credibility. Teams commonly settle into one of two bad equilibria: test only the LLM-free edges and let the main path run naked, or force exact assertions and re-run until green. The real cost lands at change time: edit one word in a prompt and generation quality can silently degrade, with no signal in CI — [part 11](/blog/deterministic-boundary/) lists this as failure mode four of a probabilistic component, "untestable regressions".

The answer our project converged on fits in one sentence: **an LLM application runs evals, not exact-value tests.** Assertions are not abandoned — they change their object, from "the output equals X" to "the output satisfies property P": structure score above a threshold, pairwise preference no worse than the previous version, a PDF that compiles, citations and numbers closed against their sources, metamorphic relations holding. Deterministic code keeps its exact assertions; the two kinds of test coexist in layers.

## The philosophy: eval-driven development

Our methodology survey (docs/testing/01-research in the project, 2026-07, compiled from multi-source web search) compresses the field consensus into three points:

1. **EDD (Eval-Driven Development)**: define the evaluation criteria before writing the agent; on every prompt/flow change → run the evals → read the score movement → decide. Evaluation is not post-hoc acceptance but the steering wheel of the development loop — TDD for agents.
2. **The behavioral-testing triad** (CheckList, ACL 2020): MFT (minimum functionality), INV (invariance), DIR (directional) tests. Transferred to this project: material with a planted gap must be flagged by the material check (MFT); deleting a data file must increase the gap count (DIR).
3. **Metamorphic testing**: when there is no reference answer, assert relations between inputs and outputs instead — "fuller material ⇒ structure score does not drop", "delete half the data ⇒ more gap warnings". This sidesteps the fundamental obstacle that non-deterministic output cannot be asserted exactly.

The tooling conclusion was **no heavy dependencies**: borrow DeepEval's form — "an eval is a pytest case plus a threshold assertion" — and wrap our own scorers into pytest cases: zero new dependencies, offline-capable, self-hosted.

## Four layers, split by "does it call a real LLM"

| Layer | What runs | Calls LLM | Speed | How to run |
|---|---|---|---|---|
| **L1 unit** | Pure logic: parsing / indexing / routing / guards / sandbox whitelist / the scorers themselves / source verification | No | Seconds | `pytest` |
| **L2 integration** | Mock LLM drives the flow: full run_agent pipeline, tool loop, endpoint contracts | Mock | Seconds | `pytest` |
| **smoke** | Real local execution: matplotlib figure sandbox, xelatex compile | No | A few seconds | `pytest` (when xelatex is present) |
| **L3 online eval** | Real keys run the agent/judge; scorers plus thresholds decide | Yes (DeepSeek + Qwen) | Minutes | `pytest --run-live` |

The layering criterion is not the textbook unit/integration split but two switches: **does it call a real LLM**, and **does it need a real execution environment**. A few design points:

- **How L2 is written**: `monkeypatch` swaps `agent._client` for a fake, tool_calls and final content are scripted, and the test asserts flow events plus an `eval_report` score — it verifies "wired correctly", not "high quality". Note the scorer reuse: the same `eval_report` scores mock output in L2 and real output in L3, while being itself an L1 test subject — the scorer is tested before it is allowed to test anything else.
- **Tests never require compilation**: `pytest` runs the Python source in seconds; the Nuitka build exists only to produce the customer-facing .exe. The dev loop = edit code → `pytest` → green.
- **Gating**: `@pytest.mark.live` is skipped by default; only `--run-live` runs it, and missing keys auto-skip. The two real keys (DeepSeek for the text/agent main path, Aliyun Qwen for vision image reading) live in `backend/.env`, never in git. CI runs only the three offline layers by default; L3 runs manually, pre-release, or nightly.

```bash
pytest                # L1+L2+smoke (default, seconds, offline, mandatory in CI)
pytest --run-live     # add L3 online evals (real keys, slow, costs money)
python run_evals.py   # one-shot golden-sample eval: scores vs threshold baseline
```

## Objective criteria: five assertable properties

Agent output is never asserted against fixed strings; these properties are asserted instead:

| Criterion | Implementation | Assertion form |
|---|---|---|
| Structure score | `eval_report.score_report`: 8 reference-free checks (compilable skeleton / ≥3 sections / error analysis / citation closure / data tables / equations / no AI boilerplate / no TODO), returns 0–1 | score ≥ threshold |
| Pairwise preference | `eval_judge.judge_pairwise`: judged twice with positions swapped, ruled only if consistent | new version no worse than baseline |
| Compile pass | Real xelatex compile | PDF produced, page count > 0 |
| Source closure | `verify.verify`: `\cite` closed against the bib, data numbers traceable to the material | no undefined citations; unsourced numbers bounded |
| Metamorphic relations | metamorphic / DIR | delete data → more gaps; fabricated number → more unsourced; fuller material → score does not drop |

Two points deserve expansion. **Pairwise preference must treat position bias**: LLM judges systematically favor the first candidate (arXiv:2406.07791), so `judge_pairwise` evaluates twice with positions swapped and rules only on agreement; before judging anything, the judge must pass a `reconstruction_accuracy` self-check — correctly ranking known strong/weak pairs (≥ 0.7) — to earn the right to judge drafts. Validity engineering for the judge itself is the subject of [part 9](/blog/judge-validity/).

**Metamorphic relations are the cheapest anti-hallucination criterion.** The DIR test in the source-verification module, verbatim (string literals translated from the Chinese original):

```python
def test_verify_metamorphic_fabricated_number_increases_unsourced():
    """DIR metamorphic: add a number absent from the material →
    the unsourced (suspected-fabricated) set must grow."""
    mat = "measured 1.0 and 2.0"
    base = verify.verify("values 1.0 and 2.0", mat)["numbers"]["unsourced"]
    more = verify.verify("values 1.0 and 2.0 and 7.77 from thin air", mat)["numbers"]["unsourced"]
    assert len(more) > len(base) and "7.77" in more
```

No knowledge of the "correct output" is needed — only a direction is asserted: one number appearing out of thin air must enlarge the unsourced set. The pattern replicates cheaply: delete a data file → more gap warnings; supply the .bib → the citation gap disappears.

## The golden regression gate: the scorer must prove itself before it may gate

Online evals are the most faithful, but they need keys and take minutes — they cannot protect *every* change. So the regression gate is dual-track: beside the online gate `run_evals.py` (real generation plus judge), an offline gate `test_golden_gate.py` runs inside plain `pytest`, no keys needed, three gates:

**Gate 1: scorer discriminative power.** Gold-standard "good/bad" reports must be clearly separated by `eval_report` (assertion messages translated):

```python
assert good["score"] >= 0.75, f"gold good report scored too low {good['score']}: {good['flags']}"
assert bad["score"] <= 0.35, f"gold bad report scored too high {bad['score']} (scorer lost discrimination)"
assert good["score"] - bad["score"] >= 0.4, "good/bad gap too small: scorer discrimination regressed"
```

The logical order matters: **first prove the scorer actually encodes quality, only then allow it to act as a gate.** A scorer whose discrimination has decayed renders every downstream threshold meaningless — and it will never raise an alarm about itself, which is why something else must pin it.

**Gate 2: the gold-standard skeleton really compiles.** The gold report skeleton must pass a real xelatex compile (skipped when no engine is installed locally; on the release machine / CI with TeX it is a hard gate). A generated report that fails to compile is the most direct signal of prompt or template regression; this gate protects the template plus the compile chain.

**Gate 3: prompt shape.** Offline tests cannot run a real LLM and cannot judge output quality — but they can judge the structure of a prompt: the scoped prompt `build_user_scoped` must still demand "rewrite only this section", with `\documentclass` and `\end{document}` appearing inside the "strictly forbidden to output" clause; the full-document path `build_user(gaps=None)` must be byte-identical to the call without gaps. The moment someone edits the prompt back to "output the whole document", plain `pytest` goes red.

(One additional foundation check: the gold-standard report must parse correctly under the deterministic sectionizer — protecting the base of section-level editing.)

## The baseline: every score pinned to git_sha and prompt_hash

The online gate `run_evals.py` outputs more than red/green: every real run appends its scores to `docs/testing/eval_baseline.tsv`, and its exit code feeds CI. The file's actual contents (fixture names translated):

| date | git_sha | prompt_hash | metric | name | score | threshold | pass |
|---|---|---|---|---|---|---|---|
| 2026-07-01 21:49 | cb5bba9 | 6f3202abff3a | judge.recon | strong/weak pair | 1.000 | 0.70 | 1 |
| 2026-07-03 11:29 | 39b9e29 | 46bed3f78ca1 | report.score | field-sweep S-params | 0.875 | 0.50 | 1 |
| 2026-07-03 11:29 | 39b9e29 | 46bed3f78ca1 | report.compiles | field-sweep S-params | 1.000 | 1.00 | 1 |
| 2026-07-03 11:29 | 39b9e29 | 46bed3f78ca1 | judge.recon | strong/weak pair | 1.000 | 0.70 | 1 |

Every row pins both `git_sha` and `prompt_hash`: when a score drifts, the first question — "did the code change or did the prompt change" — is answerable immediately. The full development loop:

```mermaid
flowchart TD
    A[Edit code / edit prompt] --> B{pytest: L1+L2+smoke, seconds}
    B -->|red| A
    B -->|green, quality-relevant code untouched| Z[Commit]
    B -->|green, agent/quality-relevant code touched| C[pytest --run-live or run_evals.py]
    C --> D[Real agent run → scorer/judge scores]
    D --> E{Compare against thresholds and eval_baseline.tsv}
    E -->|meets thresholds| F[Green: scores appended to baseline] --> Z
    E -->|regression| A
    Z --> R[Pre-release: default suite + --run-live all green → only then the Nuitka build]
```

Measured numbers from the rollout: on the day the system landed (2026-07-01), the default suite ran 144 passed + 7 skipped (live) and `--run-live` passed 7/7; two days later the golden gate was added, 248 passed with no regressions; at the time of writing the project holds 351 test functions. Flakiness governance comes down to two moves: LLM cases assert thresholds/intervals/metamorphic relations rather than exact strings, and the judge uses swap-averaging to reduce variance.

## Where this does not apply

1. **A small golden set is not a proof of quality.** The survey notes that aggregate metrics need on the order of hundreds of examples to be trustworthy (that figure not independently verified); this project's online golden set started with 1 report fixture plus 2 strong/weak pairs. At that scale the online eval is a smoke gate — it catches large regressions — not a quality metric. Do not conclude "quality improved" from it before the set grows.
2. **Objective criteria only measure the measurable.** The 8-item structure score is a reference-free floor: it judges "is this report solid and compilable", not "is it well written". Preference-level quality above the floor needs a judge — and the judge must be validity-tested first; an unvalidated judge gate is more dangerous than no gate.
3. **Green L2 is not green quality.** Mock-driven integration tests verify wiring and flow events; they know nothing about the quality of real model output. Reading L2 passes as a quality signal is the most common misreading of this layering.
4. **Online evals are slow, cost money, and carry variance — keep them out of the fast loop.** They belong pre-release, nightly, or after quality-relevant changes. And if the product is still in prototype phase with prompts rewritten daily, the threshold-maintenance cost of golden gates will exceed their value — establish a reasonably stable product definition first, then build the gates.

## References

- CheckList (ACL 2020) — the MFT/INV/DIR behavioral-testing triad; the direct source of this article's criteria design.
- arXiv:2406.07791 — systematic study of position bias in LLM judges (systematic preference for the first candidate).
- arXiv:2410.15393 — judge calibration methods: swap-and-average, balanced-position calibration, and others.
- arXiv:2504.18827 — metamorphic testing applied to LLMs (LLMorph and related work).
- DeepEval — the "eval as a pytest case + threshold assertion" form we borrowed (without taking the dependency).

These external sources come from the project's research document (2026-07, compiled from multi-source web search); apart from this project's own measurements, second-hand claims were not independently re-verified.