Engineering | kaguc — Writing to understand systems.

27 days of AI-assisted development: a retrospective in numbers

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 14 of the Engineering LLM Applications series — the closing installment. The previous thirteen articles discussed mechanisms; this one returns to the 27 days that produced them.

Scope and accounting rules

From 2026-06-10 to 07-06, one person plus a coding agent (Claude Code) built, from scratch, the AI writing agent this series has been citing throughout. Because the project enforced a paper-trail rule from day one — every development conversation produces a devlog entry — this retrospective does not rely on memory. The data comes from 94 devlogs, git history, a count of the test directory, and a line-by-line sum of the research experiment’s usage.jsonl. The article is organized as four ledgers: output, discipline, debt, and conclusions. Approximate figures state their accounting basis; anything the logs marked as “suspected cause” stays hedged here.

The output ledger

Dimension	Number	Basis
Time span	27 days	2026-06-10 → 2026-07-06
Devlogs	94 entries	docs/devlog/, one per conversation; single-day peak of 27 (06-15)
Backend	28 modules / 8,352 lines	line count over backend/app/*.py
Clients	Web frontend + Tauri desktop build	PyInstaller-frozen backend + Tauri shell + bundled TinyTeX
Tests	351	count of def test in backend/tests
Research experiment	4 days / 20 commits	writing-judgment-distill, 06-18 → 06-21
Experiment LLM usage	3,511 calls / ~7.94M tokens	summed from results/usage.jsonl: ~5.22M input + ~2.72M output
Product pivots	3	manual workflow → local web app (06-10) → SaaS (06-12) → standalone desktop (06-22)

Two notes. First, output came in bursts: of the 94 devlogs, 27 landed on 06-15 alone and 14 on 06-11 — at the granularity of “one conversation advances one feature”, parallel sprints became the norm. Second, the third pivot (SaaS → standalone) is worth recording as method: every change was flag-gated and off by default; on landing day, all 47 backend tests passed in the default configuration — zero regression in SaaS behavior. The test baseline then grew to 344 passed (devlog figures), with the repository ultimately counting 351 test functions.

The discipline ledger: which conventions paid for themselves

Of the working conventions written into CLAUDE.md on day one, four were repeatedly proven worth their cost by day 27.

A devlog for every conversation. Fixed format: goal / what was done / key decisions / files changed / leftovers. The direct payoff is this article — a retrospective that needs no recollection. The larger payoff is attribution: the venv-pollution debt described below can be reconstructed as a complete causal chain from two log entries one hour apart. The research experiment ran the same regime — journal entries are append-only and machine-readable results go into runs.tsv — which is what allowed the summary document to state that “all numbers come from recorded experiments.”

Git as the safety net: commit every state that runs. The very first conversation ended with git init and a commit (f705076, 115 files). None of the three pivots ever lost work: the standalone pivot branched from the current HEAD with zero disturbance to the working tree, preserving the experiment branch’s 20 commits intact. “If the AI breaks it, roll back” went from slogan to daily practice over the 27 days.

base_docs is read-only. Domain material, few-shot samples, and test data live in a read-only zone; generated artifacts go into separate directories. Through three pivots and dozens of generation rounds, the evidence base was never contaminated.

One feature per change. Twenty-seven devlogs in one day is not 27× productivity; it is 27 scope-narrowed conversations, paired with “one commit plus tests per feature.” This is the deterministic boundary applied isomorphically to the development process itself: narrow the scope of every change, and defend existing assets with deterministic gates — tests and git.

The debt ledger: verification debt and environment debt

Verification debt, case one: a batch verification that failed wholesale. On 07-05, to benchmark the agent pipeline against the state of the art, a 105-agent web-research workflow was run. Its adversarial verification stage failed as a whole batch when the subscription’s session quota was exhausted: all 25 claims came back unverified. The only remedy was manual: pick the 3 papers carrying the most weight — and most likely to have been fabricated by a retrieval agent — and check them line by line against the arXiv originals. One had its core numbers confirmed (revisions regress 16–27% of already-covered content); one was genuine but required two corrections to how it had been cited; one was architecturally real, with its specific numbers absent from the abstract and therefore left marked unverified. The machine produced 25 conclusions in one evening; human verification covered 3. That gap is the shape of verification debt.

Verification debt, case two: a conclusion frozen for want of a human anchor. The 4-day controlled experiment obtained highly direction-consistent results under LLM-judge scoring (see part 12); the 30-pair anonymized blind-evaluation sheet was exported, and the machine’s verdict on the same items was B=20 / A=3 / tie=7. But human scores were never collected. The summary document therefore had to settle for: H1 is “a strong candidate with method-level evidentiary support — not established.” The one step that 3,511 calls and ~7.94M tokens could not buy was precisely the only step independent of the machine.

Environment debt: one experiment’s contamination, detonating an hour later. On 06-23, minted code highlighting was trialed for the standalone build; the conclusion was that it is unworkable under frozen distribution, so the build reverted to the safe default. But the experiment’s pip installs had touched the shared venv. One hour later (by log timestamps), the packaged desktop app failed to start; one layer of the four-layer root-cause analysis was a missing pydantic_core — the log annotates it as “suspected cause: the minted detour’s pip installs overrode the pydantic version.” The recorded lesson, verbatim: clean up the venv after temporary experiments.

The methodological conclusion: the bottleneck moved to verification and judgment

The output ledger shows AI assistance expanding the engineering scale one person can drive by an order of magnitude — a full-stack application, desktop distribution, 351 tests, and a controlled experiment advancing simultaneously within 27 days. That is a case judgment against the author’s own manual pace, not a controlled measurement. But the three debts point to a single conclusion: once scale expands, the bottleneck moves from production to verification and judgment. Twenty-five claims needed a human reading the originals; H1 needed 30 human blind evaluations; the venv contamination needed a human-driven layer-by-layer diagnosis. Every conclusion the machine produces needs an anchor independent of the machine — deterministic tests, git history, arXiv originals, human scores. The mechanisms discussed across the first thirteen articles of this series are, in the end, ways of manufacturing such anchors at scale.

Where this retrospective does not apply

N=1. One developer, a greenfield project, no collaboration friction, no legacy code. These numbers are a record, not a statistic; extrapolating to teams or brownfield systems has no basis here.
Output metrics are not outcome metrics. Lines, tests, and devlogs measure production volume; the 27 days include no user data, so product value went unmeasured.
Devlog count is not workload. Devlogs count conversations; a 27-entry day reflects short, scope-narrowed sessions, not 27× throughput.
The token figures cover only the experiment’s metering. usage.jsonl records the research experiment’s API calls only; the coding agent’s own usage during main-app development is not metered there, so total cost cannot be extrapolated from these numbers.
“Zero regression” was purchased in advance. Three pivots without incident rested on flag-gating plus a test baseline; without that baseline, the same speed produces silent breakage.

References

The three papers manually re-verified after the 105-agent workflow: Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (arXiv:2601.13217; core numbers checked against the original); PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing (arXiv:2604.05018; genuine, with two citation corrections); RP-ReAct (arXiv:2512.03560; architecture genuine, specific numbers not independently verified).
Experiment data and statistical basis: the writing-judgment-distill SUBMISSION_SUMMARY and results/usage.jsonl.

Controlled revision in reflection loops

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 10 of the Engineering LLM Applications series.

The problem: why “draft → self-revise × N” often makes things worse

The standard closing act of an autonomous generation pipeline is a reflection loop: once the first draft exists, let the model critique and revise it for a few rounds. Intuition says more rounds can’t hurt. Our first implementation was a direct transcription of that intuition — critique produced a prose commentary, revise rewrote the whole document accordingly, and the new draft replaced the previous one unconditionally:

msgs.append({"role": "assistant", "content": draft})
yield {"type": "draft", "text": draft} # keep this version unconditionally

Empirical measurement says the intuition is wrong. arXiv:2601.13217 (Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision — we read the arXiv original to verify it) measured multi-turn revision in deep research agents: when handling feedback, full-rewrite revision regresses 16–27% of already-covered content on average, with citation quality degrading alongside. More alarming: unguided “please reflect and revise yourself” is ineffective for almost every agent tested — only 1 of 5 improved (+3.6%) — and it causes more collateral damage than targeted feedback. On the mechanism side, Self-Refine (arXiv:2303.17651, retrieval excerpt, not adversarially verified) is consistent: feedback must be specific and actionable, generic feedback measurably hurts, gains concentrate in the first 1–2 rounds, and tasks where errors are hard to self-detect see almost no benefit. The positive contrast is PaperOrchestra (arXiv:2604.05018): its refinement loop carries an explicit “accept only if better, revert and stop if worse” gate and reports a 79–81% win rate with 0% degradation (a body-text detail, not independently verified).

Held against that evidence, our first implementation stepped on all three mines:

Failure mode	Evidence	Corresponding rework
Full-document rewrite	Regresses 16–27% of covered content, citation degradation (2601.13217, manually verified)	Rework 1 + Rework 2
Open-ended prose feedback	Generic feedback measurably hurts; unguided self-reflection nearly useless with more collateral damage	Rework 1: locate to section + actionable step
No accept/revert gate	Only gated refinement achieves 79–81% / 0% degradation (PaperOrchestra, not independently verified)	Rework 3: zero-token regression gate

The three sections below walk through the rework in order of increasing invasiveness. The context is a LaTeX report-generation agent (scan → explore → figures → write → reflect × N → bib → compile self-repair), but the mechanisms are format-agnostic.

Rework 1: reflection outputs a structured edit plan

The first cut lands on the shape of the feedback. The critique is no longer prose; it is forced into a JSON edit plan (prompt translated here — the production prompt is Chinese):

_REFLECT_SYS = (
 "You are a reviewer of lab reports / course notes. Check the current draft "
 "against the source material and identify only problems that genuinely need "
 "fixing, locating each one to a specific section (give its anchor, e.g. n2); "
 "use \"whole\" for cross-section / overall-structure / abstract-and-keywords "
 "issues. Each item carries an issue and an actionable action. Do not mention "
 "what is already fine; no generic polishing. Output strictly JSON: "
 "{\"changes\":[{\"section\":\"n2 or whole\",\"issue\":\"\",\"action\":\"\"}]}; "
 "if the draft is acceptable overall, output {\"changes\":[]} and nothing else."
)

Four design points:

Every issue must be located to a specific section (the anchor assigned by the sectionizer, e.g. n2); cross-section, overall-structure, and abstract/keywords issues are explicitly marked "whole". Location is not decoration — it decides which application path the item takes next.
Issue and action come as a pair. “The error analysis is shallow” alone is not enough; an executable action is required — the direct application of Self-Refine’s “feedback must be specific and actionable”.
The empty array is allowed. {"changes":[]} means “acceptable overall” and terminates the reflection loop early. Revision flips from “change by default” to “no change by default, change only with cause”.
Parse failure never breaks the flow. _parse_reflect_plan returns an empty list for non-JSON or missing changes, which the caller treats as “no changes needed”; plans are capped at 8 items; the title fallback when an anchor misses is “exact match first, then the longest contained title” — so “Results” cannot steal a match meant for “Results and Discussion”.

A companion change: reflection now runs on a fresh minimal context (system + source material + current draft) instead of dragging along the full exploration transcript; the material sits as a stable prefix and hits the cache across rounds.

Rework 2: pure section-level issues go through scoped per-section splicing

Each plan item is routed by its location. Pure section-level issues no longer trigger a full rewrite; they reuse the mature machinery of the interactive editing side (sectionizer, scoped prompts, fragment health check, deterministic splice — the toolkit of Part 07), applied section by section. For each item, _scoped_reflect_apply re-parses and re-resolves the target on the current working text (offsets have moved after the previous splice), sends the model only the target section plus that item’s issue/action plus relevant material, and health-checks the returned fragment (non-empty, balanced braces, balanced environments, no \documentclass). A fragment that fails the check causes that section to be skipped — a bad fragment is never spliced back. A fragment that passes is spliced in, with every other section byte-identical. Only cross-section issues, abstract/keywords issues, or drafts that cannot be sectionized fall back to a whole-document minimal rewrite.

This step has one implementation pitfall, found by an internal adversarial audit (rated [medium]):

# The fragment must preserve the count of top-level \section commands within the
# replaced span (usually =1; =0 for a subsection) — otherwise the splice changes
# the document's section count, the positional anchors of subsequent scoped items
# in the same round drift and edit the wrong section, and the "added sections"
# direction is not caught by regression_check (it only reports decreases).
# Count mismatch → skip this section; never splice a fragment that would shift things.
if len(re.findall(r"\\section\{", new_text)) != len(re.findall(r"\\section\{", section_src)):
 continue

The model occasionally gets “helpful” and splits one section into two: the fragment itself is healthy and passes every check, but after splicing, the document’s section count goes up by one, the positional anchors of all subsequent plan items in the same round drift, and the wrong sections get edited — while the regression gate only reports section decreases, leaving this path with no backstop. The conservation check closes it off before the splice.

Rework 3: the zero-token regression gate

Whether a round went through scoped splicing or the whole-document fallback, its output passes regression_check before landing — six criteria, all deterministic, zero API calls:

loss of the compilable skeleton (\documentclass / \end{document});
length collapse below 60% of the previous version (truncation, or “rest unchanged”-style elision);
structural score decrease;
shrinkage of the \cite key set (with the lost keys reported);
fewer sections;
fewer figures or data tables.

Any trigger means regressed: revert to the previous version and stop reflecting. If the two versions are character-identical after whitespace stripping, the result is converged — also an early stop, so convergence wastes no rounds.

Why not use an LLM judge as the gate? The reason is written in the module’s own comments: this particular failure mode — losing content — is deterministically detectable, whereas the pairwise judge’s reliability in the Chinese domain is unverified (the judge module’s own caveat) and would cost one extra real call per round. More correct, and cheaper. The judge’s proper place is offline evaluation, not an online gate; the validity question is the subject of Part 09.

The gate has its own edge case: in the pipeline, bibliography generation runs after reflection, so during reflection the bib is always empty — which makes the “citations closed” check false for any draft containing \cite, and a newly added citation would be misread as a structural-score drop, wrongly reverting a good edit. The fix: while the bib is empty, that check is excluded from the gate score, and citation loss is judged separately by “the key set must not shrink” — pinned by a dedicated regression test (empty bib produces no false positive).

Gate decisions are surfaced as persisted step events (visible in the log page and across reconnects), with a structured reflect_gate field attached to the trace so the experiment bench can observe whether reflection is actually improving quality and which round reverted.

The reworked loop, end to end

flowchart TD
 A[Current draft vN] --> B[Reviewer on fresh minimal context: JSON edit plan]
 B -->|changes empty| S[Judged acceptable, early stop]
 B -->|pure section-level| C[Scoped per-section edit + splice, other sections byte-identical]
 B -->|contains whole items| D[Whole-document minimal rewrite]
 C --> G{Zero-token regression_check}
 D --> G
 G -->|degraded| X[Revert to vN, stop reflecting]
 G -->|converged| S
 G -->|pass| V[Commit as vN+1, next round]

Tests arrived in batches with the rework: when the gate landed, test_eval_report +7 (accept / revert / converge / lost section / lost citation / length collapse / empty-bib no false positive) and test_run_agent_mock +1 (degraded revision → revert to previous version and stop), with 303 backend tests green; 306 after scoped reflection landed; 322 by the end of this round of work (plus 19 on the experiment bench). Generation quality itself cannot be tested offline — but every deterministic component of the loop can: plan parsing, routing, splicing, the gate.

Lessons from the research process itself

The evidence chain behind this rework deserves its own post-mortem. To benchmark against the state of the art, we ran a 105-agent networked research workflow; the adversarial-verification stage failed wholesale when the subscription session quota was exhausted, leaving all 25 claims stuck at unverified. So we manually re-verified the 3 most load-bearing papers — the ones a retrieval agent would most plausibly fabricate — by reading the arXiv originals. 2601.13217: real, core numbers confirmed. PaperOrchestra: real, but with two corrections — it is not a Google paper, and its abstract describes itself as flexible rather than rigidly fixed, so it can only support “structured multi-agent division of labor beats autonomous loops” (win rate +50–68% / overall quality +14–38%), not “the more rigid the better”. RP-ReAct (arXiv:2512.03560): the architecture is real, but the specific scores in the retrieval claim about “being beaten by vanilla ReAct on simple tasks” do not appear in the abstract — plausibly extracted or confabulated from the body text by the retrieval agent, so they are marked unverified. One more calibration correction: the retrieval claim said “break rate 21–31%”, which is the body-text framing; the abstract says 16–27%, and this article uses the abstract’s figure.

The lesson is isomorphic to the article’s thesis: a retrieval agent’s output needs an accept/revert gate of its own — manual verification is the regression_check of the research workflow.

When not to do this

The gate’s criteria are proxies: they prevent getting worse, they do not certify getting better. Section counts, citation sets, and length all measure structural preservation; content that is wrong but structurally intact passes the gate. Preference-level quality still requires offline evaluation and judges — this gate does not replace them.
Scoped reflection presupposes sectionizable structure. A draft that cannot be parsed goes straight down the whole-document fallback — that fallback is not a defect, it is an acknowledgment of the boundary.
When the structure itself is wrong, scoped editing is powerless. Problems requiring global reorganization (moving sections, merging sections, changing the narrative line) must be marked whole and take the full rewrite; forcing them through per-section edits produces drafts that are locally fluent and globally incoherent.
Do not bolt this gate onto loops that already have an external signal. The compile self-repair loop has the compiler as its referee and stops on no-progress error fingerprints — a different kind of closed loop, and already the right design; adding a structural gate to it is redundant construction.
Keep N small. Self-Refine reports gains concentrated in the first 1–2 rounds (not adversarially verified), and 2601.13217 shows the break rate is the bottleneck of multi-turn revision — budget spent on a fifth reflection round is most likely paying for a revert.

This article pairs with Part 11, “The deterministic boundary”: that article promotes the gate and the scoped machinery into a unified architectural principle; this one is the complete rework record of that principle applied to a single closed loop — the reflection loop.

References

Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (arXiv:2601.13217) — full-rewrite revision regresses 16–27% of covered content on average with citation degradation; unguided self-reflection is nearly useless and causes more collateral damage; its named remedies: structured edit plans and a dedicated Reviser. Verified against the original.
PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing (arXiv:2604.05018) — structured multi-agent pipelines beat autonomous baselines by +50–68% / +14–38% (verified); its accept-or-revert refinement gate reports a 79–81% win rate with 0% degradation (body-text detail, not independently verified).
Self-Refine: Iterative Refinement with Self-Feedback (arXiv:2303.17651) — feedback must be specific and actionable, generic feedback measurably hurts; gains concentrate in the first 1–2 rounds; tasks where errors are hard to self-detect need external signals. Retrieval excerpt, not adversarially verified.
Reason-Plan-ReAct (arXiv:2512.03560) — plan/execute decoupled architecture (verified); the specific scores for “beaten by vanilla ReAct on simple tasks” are absent from the abstract and unverified.

Distilling judgment, not structure: a controlled experiment

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 12 of the Engineering LLM Applications series.

The hypothesis: can distilled judgment beat a distilled template?

Hand expert exemplars to a model and ask it to induce a reusable writing skill (a prompt), and there are two routes:

Path A (structural template): extract “how many parts, what goes into each, in what order, in what format”;
Path B (judgment logic): contrastive rationale extraction — show the model strong and weak versions of the same task and force it to articulate why the strong one is better, what it assumes about the reader, how a novice gets it wrong; aggregate the output into judgment rules + do/don’ts + a reader model.

H1: the Path-B skill beats the Path-A skill in pairwise preference judging. Everything is inference-only — no training, no fine-tuning, LLM APIs all the way down. Most prompt-engineering practice defaults to Path A; if H1 holds, there are many places where the recipe should change. This article is a full experiment report, including three methodological corrections that each came close to voiding the conclusion.

Design: control everything except one variable

All data comes from public benchmarks. Exemplars and strong/weak pairs are drawn from ASAP-AES 2.0 (English argumentative essays, human holistic scores 1–6): exemplars are score-6 essays; strong/weak pairs contrast score 5–6 against score 1–2 whole essays. A further 120 human-scored essays (20 per band) serve as a judge-credibility anchor, and the ablation uses arXivEdits same-sentence before/after expert rewrites.

The control discipline: A and B share the same exemplar input, the same generation model (DeepSeek-v4-pro, T=0.7), the same tasks, the same length constraint (roughly 250–400 words, no padding), and the same judges — the only variable is the distilled skill. Three bias controls on the judging side: position swapping (a win counts only if both presentation orders agree, otherwise it is a tie), identity masking (judges see only “Essay One / Essay Two”), and length neutrality (the rubric explicitly ignores length and penalizes verbosity). Dual judges: Claude-Sonnet-4.6 (primary) + Qwen3.7-max (cross-check).

The two distilled prompts are distinguishable at a glance. skill_A is position-and-proportion instructions throughout; a typical entry:

Introduction (1 paragraph) … A clear thesis statement that takes a definite position. … Relative length: 3–5 sentences.

Every entry in skill_B carries a “for whom, and why”:

When you acknowledge a counterargument, respond with a genuine rebuttal – not dismissal – because your reader expects fairness and intellectual seriousness.

Statistical lesson 1: effective N ≈ 1

The first round distilled a single pair (skill_A, skill_B), generated on 30 tasks, judged pairwise, and ran a binomial test treating the 30 pairings as independent samples: p ≈ 7×10⁻⁹. A beautiful number built on a broken inference: those 30 pairings share one and the same skill pair — only the tasks vary. What the test measures is “how stable this one skill pair is across 30 tasks”. H1, however, is a claim about methods, and the independent unit for that claim is the skill — of which there was exactly one pair. Effective N ≈ 1.

The redesign: first expand the exemplar pool from 12 to ≥24 essays, then use distinct random seeds to draw 6-exemplar subsets (Path B draws matching strong/weak-pair subsets), distilling K=10 independent skill pairs. Each pair generates and is judged on a shared task set (15 pairings per pair entered the statistics as executed), yielding one preference score w_k per pair; the test then runs at the skill level — Wilcoxon signed-rank on whether the K values of w_k are systematically above 0.5, with confidence intervals from a cluster bootstrap that resamples by skill. At the redesign’s budgeted scale this comes to 1200+ LLM calls across generation, judging, and distillation — roughly 5–6× the first round. Statistical honesty costs money.

flowchart LR
 P[Exemplar pool ≥24] -->|seed k draws 6 exemplars + pair subset| D[Distill K=10 pairs<br/>skill_A_k and skill_B_k]
 D --> G[Shared tasks ×2 generations per pair<br/>same model · T=0.7 · 250–400 words]
 G --> J[Dual-judge pairwise eval<br/>position swap · masking · length-neutral]
 J --> W[One preference score w_k<br/>per pair per judge]
 W --> S[Skill-level Wilcoxon<br/>+ cluster bootstrap CI]

Statistical lesson 2: metrics degenerate silently

A measurement check before the redesign saved the entire test: across 55 decided pairings under two judges and two rounds, A had won zero times — all the variation lay between “B wins” and “tie”. Define w_k = B/(B+A) as intuition suggests, and with A ≈ 0 almost every w_k collapses to exactly 1.0 — K samples with zero variance, and the Wilcoxon test degenerates into measuring nothing. The fix is a tie-inclusive preference score:

w_k = (B wins + 0.5 × ties) / T

Ties pull w_k toward 0.5 and restore real variance: on the original single pair this measures ≈0.97 under Claude and ≈0.73 under Qwen — the effect-size gap between the two judges is preserved and becomes testable.

Results: direction rock-stable, effect size judge-dependent

Judge	w_k mean	median	std	95% CI (cluster bootstrap)	w_k>0.5	Wilcoxon two-tailed p	old “per-pairing” p (overestimate, do not cite)
Claude (primary)	0.830	0.833	0.143	[0.747, 0.913]	10/10	0.0020	7.95e-24
Qwen (cross)	0.617	0.617	0.086	[0.567, 0.667]	9/10	0.0039	2.06e-06

Four notes that must be read alongside the table:

The old per-pairing p values are flagged in the statistics script’s own output: “overestimate: K×T pairings are non-independent; shown for contrast only, do not cite”. Landing a known-misleading number on disk together with its warning is more reliable than clarifying it verbally afterwards.
p=0.0020 is a floor, not an effect size. When all K=10 pairs point the same way, the exact two-tailed Wilcoxon p bottoms out at 2/2¹⁰ ≈ 0.0020 — equivalent to a sign test. It encodes only “10 pairs, one direction”, not by how much: w_k values of all-0.51 and all-0.99 yield the identical p, and p<0.001 is unreachable before K≥17. Effect-size evidence lives in the CIs.
Qwen’s “significance” carries less weight: tie-dominated, swap-consistency mostly 0.2–0.5, CI lower bound 0.567 hugging 0.5. But a decided-only sensitivity analysis (ties removed) still shows 9/10 above 0.5 with zero net losses — the significance is not manufactured by tie-weighting.
A is not winless: A won individual pairings in 4 pairs under Claude and 8 under Qwen — but B suffered a net loss in no pair under either judge.

Ablation: does the judgment signal survive same-sentence pairs?

B’s contrast source is cross-topic whole-essay pairs — its wins could be contaminated by topic/content differences rather than pure “judgment”. Swapping the contrast source for arXivEdits same-sentence rewrite pairs (near-zero topic difference) yields B_sent, run over the same 10-pair protocol:

Judge	B_sent w_k mean	95% CI	pairs >0.5	vs A (Wilcoxon p)	B vs B_sent
Claude	0.720	[0.663, 0.773]	10/10	0.0020 (significant)	+0.110, p=0.186 (n.s.)
Qwen	0.567	[0.493, 0.640]	6/10	0.109 (not significant)	+0.050, p=0.281 (n.s.)

The reading stays conservative: under Claude, B_sent still significantly beats A and is statistically indistinguishable from B, suggesting the advantage stems from judgment rather than topic. But Qwen does not reproduce the ablation, and a genre confound remains — arXivEdits are scientific-paper sentences while the generation task is argumentative essays, so “same-sentence” and “genre” changed simultaneously; the clean single-factor control has not been run.

Judgment purity: an adversarial audit revises the self-assessment down

One question could still overturn the whole narrative: are skill_B’s entries genuine judgment rules, or structural descriptions in new clothing? LLM self-assessment cannot be trusted — the pilot’s sharpness self-check reported 100%. Instead, an 11-agent adversarial audit (executed by Claude Opus 4.8: classify every entry as judgment/structure, cross-compare against the paired skill_A for re-skinning, and specifically attack “is the because a tautology?”) spot-checked 5 pairs:

Pair	Initial self-ratio	After adversarial review	Overturned entries (re-skinning suspects)
1	87%	80%	context-in-opening ≈ A’s Intro recipe
2	82%	73%	“vivid example” ≈ show-don’t-tell tautology
3	73%	67%	ordering by persuasiveness ≈ structural organization
4	82%	64%	rebuttal / closing ≈ A’s counterargument / conclusion sections
5	100%	90%	point ordering = an organization rule

After adversarial review the mean drops to 0.746, yet all 5/5 pairs remain judgment-dominated and substantively distinct from skill_A (which is position/format instructions throughout, with zero reasoning). Verdict: “pass (with conditions)” — 0.75 is an upper bound that includes redundant clusters, and pair 4 is the weakest link (0.64). The overturned entries share one pattern: a structural action glued to a near-vacuous because. Note that this remains automated AI review, not independent human verification. (Adversarial review as an engineering practice is the subject of Part 13, Multi-agent adversarial review as an engineering practice.)

Seven honest limitations

No human anchor collected (the most fatal). A 30-pair blind-evaluation list has been exported (the machine ruling on the same pairings: B=20 / A=3 / tie=7), but human scores are not yet in. Until then, every piece of preference evidence comes from LLM judges, and a systematic LLM-judge preference cannot be ruled out.
Judge–distiller family overlap. The primary judge, Claude-Sonnet, shares a model family with the Claude Opus that performed skill distillation and the purity audit; every “strong” result happens to come from that same-family judge — a self-preference risk.
p bottomed out; effective N still optimistic. p=0.0020 encodes only “10/10 same direction”; and the 10 subsets, drawn from a finite pool, necessarily overlap — “effective N=10” is not strictly i.i.d. and remains an overestimate.
Effect size is judge-sensitive. Claude 0.83, Qwen 0.62: “B does not lose to A” is solid; “how much better B is” is not settled.
Judge range collapse. QWK=0.456 (Spearman 0.813), and the confusion matrix shows the judge never awards a 5 or 6 — discrimination at the high end is unverified, precisely the region where B vs A (“both essays are decent”) is being compared. The systematic treatment of judge validity is Part 9, Validity engineering for LLM-as-judge.
Single-judge ablation + genre confound. The same-sentence ablation is significant only under Claude, and it mixes in the scientific-sentence → argumentative-essay genre variable.
Mechanism and external validity. The secondary signal — rhetorical move coverage, A 0.89 vs B 0.86 (a weak negative conclusion: no evidence that B wins by stacking moves) — comes from the pilot, was not re-measured in the main experiment, and carries no statistical test; and only one generation model and one genre were tested.

The conclusion therefore freezes as: under the LLM-as-judge lens, “learn judgment > learn structure” receives preliminary support that is method-level, directionally highly consistent, and purity-checked (with conditions); H1 is a well-evidenced candidate, not an established result. Establishing it requires human evaluation, a third-party judge, and a clean same-sentence same-genre ablation.

When not to do this

If a deterministic metric exists, skip the pairwise judges. This heavyweight statistical apparatus is only warranted where quality can only be measured by preference comparison; wherever an assertion can be written, one line of assertion beats 800 judging calls.
Path B presupposes strong/weak pairs. Contrastive distillation extracts rationale by confronting strong and weak versions; in domains with no natural quality banding (such as human scores) and no credible synthetic degradation, Path B has nothing to contrast.
If the budget cannot afford K pairs, do not claim a method-level conclusion. Single-pair results support only “B beat A on this pair”; treating the task count as the sample size was this experiment’s earliest mistake — and the easiest one to make.
Structural templates are not useless. Path A matches B on rhetorical move coverage (0.89 vs 0.86); when the task’s bottleneck is format compliance rather than quality judgment, Path A is cheaper and steadier.

Document versioning and scoped editing

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 7 of the Engineering LLM Applications series. The evidence base is the same as in earlier parts: the implementation, devlogs, and test suite of a production-grade AI writing agent (FastAPI + React + Tauri).

The problem: the model edits the document — what guards the document?

A typical incident from live testing: the user types feedback like “polish this”, “fix the bibliography”, “change Table 1” into the chat box, and some of these silently rewrite the entire document — the result lands directly as a new version, and the user discovers only afterwards that the change went far beyond what they asked for. In this class of application, the document is the user’s core asset, and every model output can touch it. Translating the problem from “model behavior” into “data model”, the requirements converge on four:

Every version recoverable — the state before any rewrite can be brought back;
Every edit rejectable — model output is a proposal first, and becomes fact only when the user accepts;
Every edit addressable — “change only this section” requires a deterministic answer to “which bytes are this section”;
Review cost bounded — the user must be able to see what changed before deciding.

None of these can be guaranteed by prompting. The rest of this article walks through our implementation layer by layer: the version chain, the proposal ledger, the sectionizer, and the division of diff labor.

The version chain: restore forks, never overwrites

The storage layer is one document_versions table (SQLite, single-user local application):

CREATE TABLE IF NOT EXISTS document_versions (
 id INTEGER PRIMARY KEY AUTOINCREMENT,
 session_id INTEGER NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
 version_no INTEGER NOT NULL,
 content TEXT NOT NULL, -- LaTeX source
 change_summary TEXT,
 parent_version_id INTEGER,
 ...
);

Three design points:

Full snapshot per version, no deltas. content stores the entire source. The cost is storage redundancy; the payoff is that reading any version requires zero replay and both ends of any diff are always materialized. For LaTeX documents in the tens-of-kilobytes range, this trade-off is not close.

version_no is numbered per session. add_version takes MAX(version_no)+1 within the session: the user always sees this document’s v1/v2/v3, never a global v847; the global autoincrement id exists only for foreign keys and lineage. A dedicated test pins this: interleave versions across two sessions, and the second session’s first version is still 1.

parent_version_id records lineage; restore = fork. The restore endpoint in its entirety:

@app.post("/api/versions/{version_id}/restore")
def restore_version(version_id: int, ...):
 """Restore/fork: create a new latest version from a historical
 version's content (parent points at it)."""
 v = db.get_version(version_id)
 ...
 ver = db.add_version(v["session_id"], v["content"],
 f"restored from v{v['version_no']}", v["id"])

Restore deletes and overwrites nothing: it creates a new version from the restored version’s content, with parent pointing at the restored version. The test assertion nails the semantics down: restoring v1 after v2 exists produces v3 — content equals v1’s content, version_no only ever increases, parent_version_id points at v1. “Undoing a restore” is just another restore; every operation is an append.

There is no UPDATE content path anywhere in the backend; manual edits also go through add_version. Immutability is the foundation everything else stands on: diffs get stable endpoints, and proposals get a well-defined base.

The proposal ledger: edits land in the ledger first, the tree only on accept

The first implementation used “plan A”: candidates were not persisted. edit-section computed but did not store; the frontend previewed the diff and called the save endpoint itself — zero new tables, no contact with the version tree. Its limits showed quickly: a page refresh lost the candidate, multiple candidates could not coexist, and the server had no audit trail. Hence the upgrade to “plan B” — the proposed_edits ledger:

CREATE TABLE IF NOT EXISTS proposed_edits (
 session_id INTEGER NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
 base_version_id INTEGER NOT NULL, -- which version the edit is based on
 char_start INTEGER NOT NULL,
 char_end INTEGER NOT NULL,
 old_text TEXT NOT NULL DEFAULT '',
 new_text TEXT NOT NULL DEFAULT '',
 status TEXT NOT NULL DEFAULT 'pending', -- pending/accepted/rejected
 scope TEXT NOT NULL DEFAULT 'section', -- section/whole/bib
 ...
);

Every edit the model produces — section-level, whole-document (scope='whole', range [0, len]), bibliography — first becomes a pending candidate; the version tree is touched only on accept, and accept reuses the same add_version:

flowchart LR
 A[Edit request] --> B{Scope}
 B -->|section / bib| C[edit-section]
 B -->|whole| W[edit-whole]
 C --> P[proposed_edits: pending]
 W --> P
 P -->|reject| X[rejected: version tree untouched]
 P -->|accept| G{base still latest?}
 G -->|no| E[409 candidate stale]
 G -->|yes| S[splice] --> V[add_version: new version]

The accept path carries two 409s: candidate already processed (accepted/rejected cannot be re-processed), and base is stale (base_version_id no longer the latest version). The second is the critical one: a candidate’s character range is computed against its base version; splicing against a base that is no longer the latest would silently drop every change between base and head. An adversarial review had found the same failure on the frontend — editing on an old version and accepting would overwrite subsequent versions — the frontend gained an “edit only on the latest version” gate, and this 409 is the server-side backstop.

Whether every edit should require confirmation got its own round of sourced interaction research. The conclusion first corrected the risk framing: a full rewrite is in fact already revertible (the version chain exists); the real risks of a silent default are scope surprise, cost, and expensive after-the-fact review — not data loss. So the confirmation gate should be light (a banner plus a plan card), not blocking modals everywhere; small section-level edits keep one-click accept. An industry counter-example is on record too: Cursor weakened its diff-approval-style confirmation and the community treated it as a regression (taken from our research notes, not independently verified). The direction matches the HITL literature: confirm only irreversible actions and prefer undo (NN/g); when unsure, narrow the scope of action and disambiguate before acting (Microsoft HAX G10).

The deterministic sectionizer: equal-length masking and dual anchors

Where do a candidate’s char_start/char_end come from? A deterministic sectionizer (~260 lines, zero LLM calls, zero third-party dependencies) parses LaTeX source into a node tree along \section / \subsection / \subsubsection. A node’s editable span runs from its heading command to the next heading whose level is not deeper than its own — editing a subsection replaces only that subsection; editing a parent section takes its children with it. Three design points:

Mask, don’t shift. A \section inside a comment or a verbatim environment (verbatim / lstlisting / minted, …) is a false heading; deleting those regions before parsing would destroy every character offset. The fix replaces them with equal-length whitespace:

def _mask(content: str) -> str:
 """Replace comments and verbatim environments with equal-length spaces
 (all character offsets preserved), so a \section inside them is not
 treated as a real heading — while splice still uses original offsets."""

Headings are located on the masked text; titles and content are read from the original at the same offsets. Parsing and splicing share one coordinate system — no translation step, no translation bugs.

Dual anchors. Each node carries both a positional anchor (n0, n1, … in order of appearance; re-parsing the same content always yields the same result) and a content-fingerprint key (L1:ErrorAnalysis, derived from level + title, with an ordinal suffix when titles repeat). Positional anchors drift when sections are inserted or removed; content keys follow the title and are more stable across versions, so they take precedence during resolution. Abstract and keywords have no \section to address, so pseudo-nodes are constructed from the \abstract{...} / \keyword{...} commands — “fix the abstract” also gets scoped editing instead of collapsing into a full rewrite.

Parse failure degrades, never raises. If no section structure parses, the sectionizer returns editable=False and the caller falls back to whole-document revision. The sectionizer’s job is to narrow scope, not to add a failure point.

The robustness of this layer was largely ground out by adversarial review; three real defects, each now pinned by a regression test: the optional-argument heading form \section[short]{long} was missed (the section merged into its predecessor and was silently lost on accept); \section inside comments and verbatim environments produced ghost nodes (corrupting the splice); and the stale-version overwrite problem described above.

Splice, fragment health checks, and a lesson from the reflection loop

A model-returned section fragment travels back into the master document through three all-code steps: strip_fragment cleanup (remove Markdown fences; if the model disobeyed and returned a full document, cut out the body — the general approach to output cleanup is Part 3 of this series); check_fragment health checks (balanced braces, balanced environments, no \documentclass — soft signals that warn rather than block); and finally a one-line splice:

def splice(content, char_start, char_end, new_text):
 return content[:char_start] + new_text + content[char_end:]

Bytes outside the range are unchanged — that promise is written directly into the test assertions.

One lesson deserves its own paragraph. The autonomous reflection loop’s scoped splicing used positional anchors: if a fragment carried one extra top-level \section, the section count went up by one after splicing, the positional anchors of every subsequent edit in the same round drifted, and the wrong sections got edited — while the regression gate of the time only caught “fewer sections”, not “more”. Silent mis-edits. The fix adds a conservation law to the splice: a fragment must preserve the count of top-level \section commands within the replaced span, otherwise that section is skipped. This too came out of adversarial review, and is now a permanent test.

The division of diff labor: the backend guarantees invariants, line-level rendering goes to the frontend

With versions and candidates in place, “see what changed” was deliberately not built as a backend service:

Layer	Responsibility	What it does not do
`document_versions`	Immutable versions, queryable lineage	No diff computation
`proposed_edits`	`old_text` / `new_text` archived as a pair	No presentation
`split_aligned`	Split a whole-document change per section when skeletons align	Returns `None` on structural change
Frontend `diffLines` (jsdiff)	Line-level highlight rendering	Never mutates data

split_aligned is the only place the backend “computes a diff”, and its conditions are deliberately tight: the top-level \section skeletons of base and new must have the same titles, same order, same count, and the preamble must be byte-identical — only then is the whole-document change split into a list of independently acceptable per-section candidates. When skeletons align, the section ranges are non-overlapping, so accepting one section replaces only that section and leaves everything else byte-for-byte untouched. If any condition fails, it returns None and the change falls back to whole-document all-or-nothing. The preamble check looks strict, but the reason is concrete: if a change lands outside every section, per-section review would miss it.

Line-level diffing goes to jsdiff’s diffLines on the frontend. Display granularity is a UI concern that iterates with the design; the backend promises only testable invariants — immutable versions, bytes-outside-range unchanged, and the safety conditions for splitting.

Quantified: how tests pin these semantics down

Semantics	The test / data that pins it
Restore = fork	After restore: `content` equals the source version, `version_no` only increases, `parent` points at the restored version
`version_no` per session	Interleaved versions across two sessions; the new session’s first version is still 1
Candidate lifecycle	409 on re-processing; 409 on stale base
Sectionizer robustness	29 P0/P1 tests (including 4 added after review); full suite at the time: 173 passed
Per-section review	6 new `split_aligned` tests; full suite at the time: 266 passed
Ownership isolation	Non-owners get 404 on version reads and restore

All of these run with a mocked LLM, fully offline — the data-model layer is precisely the part of an LLM application that conventional assertions can test to death.

Where not to do this

Full snapshots have a ceiling. Storing the whole document per version is right for tens-of-kilobyte LaTeX; at megabyte scale or high save frequency (real-time collaborative editing), switch to delta storage or CRDTs — this model is not designed for that regime.
The ledger is for human-in-the-loop products. If the product is a fully automated pipeline where nobody reviews candidates, the ledger is just an unread queue; what belongs there instead is a deterministic regression gate before anything lands (see Part 11). Likewise, wrapping every low-stakes edit in confirmation is the “crying wolf” anti-pattern NN/g warns about — we keep one-click accept for small section edits and place the confirmation gate only on whole-document changes.
The sectionizer presupposes parseable structure. On unstructured text, editable=False: the version chain still works, but scoped editing degrades to whole-document revision.
split_aligned’s conservatism is a feature, not a defect. When the skeleton changes it refuses to split: structural changes should not be accepted piecemeal — a half-accepted reorganization is more dangerous than a whole-document replacement.
The version tree is not a branching workflow. Lineage is fully recorded, but the presentation is linear with a single tip; exploring several directions in parallel and merging them is unsupported. For git-like branch semantics, this model provides only the foundation.

References

The following sources come from the sourced research notes (devlog) of our interaction study:

Eric Horvitz, Principles of Mixed-Initiative User Interfaces (CHI ‘99) — the principled framework for mixed-initiative interaction.
Microsoft HAX Guidelines — G9 (support efficient undo), G10 (scope services when in doubt / disambiguate before acting), G16 (convey the consequences of actions).
Nielsen Norman Group guidance on confirmation dialogs — confirm only irreversible actions, prefer undo, avoid crying wolf.
Google PAIR, People + AI Guidebook — users resist full autonomy on high-control artifacts; explanation serves understanding.
Cursor community regression feedback on weakened diff approval — taken from our research notes, not independently verified.

Executing model-generated code: a two-layer sandbox

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 5 of the Engineering LLM Applications series.

The problem: why model-written code must be executed at all

One class of content in automated reporting cannot avoid code execution: experiments where only raw data exists — .mat field-sweep files, csv band structures — while the existing figures are MATLAB .fig files that cannot be embedded in a PDF. The LLM can read the data format but cannot draw: turning data into a figure requires computation plus rendering, and only code does that. So the design is the same as Code Interpreter: the LLM writes a piece of matplotlib code, the backend executes it in a subprocess, out comes a PNG, and compilation embeds it into the report.

On the agent side this is a single tool, make_figure, with exactly two parameters: {name, code}. The agent calls it from its tool loop once it understands the data; on failure it receives the error and may revise the code and retry; on success the report references the figure via \includegraphics{用图/<name>}. The system prompt sets the rule alongside: if there is data but no figure, plot it — never fabricate images.

The cost is equally direct: this executes arbitrary Python generated by a probabilistic component on the backend — a built-in RCE entry point. This article dissects the two-layer sandbox designed for it, and the measured results.

Threat model: the model cannot tell data from instructions

The attack chain is clear: an untrusted data folder → file contents enter the prompt → prompt injection → the LLM writes malicious code → the backend executes it. The root cause is that an LLM processes data and instructions in the same token stream (in-band signaling) — structurally the same defect as SQL injection (data mixed into queries) and buffer overflows (data mixed into control flow). It cannot be fixed at the model layer: alignment training lowers the probability but offers no guarantee. The industry’s consensus path is therefore to assume the injection will succeed and control the damage.

The engineering corollary: the security design must not depend on the assumption that “the model won’t write bad code.” The gates must sit after the code leaves the model and before it has any effect. We built two layers: static interception before execution and runtime isolation — the first blocks known-dangerous patterns before they run, the second assumes the first will leak and minimizes what leaked code can do.

Layer 1: the AST allowlist — static interception before execution

ast_check parses the model’s code into an AST (a parse failure is rejected as a syntax error) and walks every node; any violation rejects the whole submission:

Import allowlist (17 modules): only data/plotting modules — numpy/scipy/matplotlib/pandas/math/csv/json and the like — with both import and from ... import compared by top-level package name. The stance is allowlist, not blocklist: the criterion is “what does a plotting task need,” not “what would an attacker use.”
Dangerous-name blocklist (37 names, checked on bare Name nodes, not just imports): os/sys/subprocess/socket/ctypes/importlib/pickle/pathlib/glob… — any appearance rejects. This rule is not redundant: the execution wrapper pre-imports os to implement file navigation, so model code can call os.remove without ever writing import os. Usage must be intercepted, not merely importing.
Forbidden builtins (17): eval/exec/compile/__import__/open/getattr/globals… — banned both as calls and as bare-name references, so aliasing like e = eval is rejected too.
Dunder escapes: any __xx__ attribute access is rejected, closing the classic ().__class__.__subclasses__() escape chain:

elif isinstance(node, ast.Attribute):
 if node.attr.startswith("__") and node.attr.endswith("__"):
 return False, f"禁止访问 {node.attr}"

Rejection is not a dead end. The error returned to the agent is instructional: it states which libraries are allowed, that data is read via srcpath('filename'), that output is saved via plt.savefig(OUT), and what is banned. The agent takes this deterministic signal, rewrites, and retries — the same closed-loop pattern as compile self-repair: feed the output of a deterministic check back to the model.

Layer 2: runtime isolation — nothing left to damage after a leak

Layer 1 is static analysis over text and can, in principle, miss. Layer 2 assumes it already has, and shrinks the execution environment to the minimum:

Measure	Damage it targets
Environment built from scratch: only the allowlisted variables PATH/MPLBACKEND/MPLCONFIGDIR/HOME/LANG	API keys and proxy settings drop out automatically instead of leaking via the environment
`_GUARD` preamble injected before user code: `os.system` replaced with a lambda raising PermissionError, `sys.modules['subprocess']` set to None, `socket.socket` disabled	Command execution and network egress after a static miss
Isolated cwd: a fresh `mkdtemp` directory per run, deleted afterwards	Writing into host directories
`python -I`: isolated mode	Injection via PYTHONPATH / user site-packages
90-second timeout	Infinite loops and resource exhaustion
Source-data directory mounted read-only by compose (container deployment)	Tampering with source data

Building the environment from scratch has a platform cost: on Windows, a CPython subprocess needs SystemRoot and related system variables (crypto RNG, system DLLs), and matplotlib needs a writable MPLCONFIGDIR/TEMP — these must be explicitly back-filled, or figure generation fails unconditionally in the Windows distribution.

There is also an easily missed companion design: every ban must ship with a replacement. The allowlist bans os/glob/pathlib, so the model cannot list directories or join paths — offering no substitute would force it into violations. The wrapper therefore injects three navigation primitives, all confined to the source-data directory SRC: srcpath('relative-name') for paths, listdir(sub='') for directory listings, and walkfiles() for a recursive list of all relative filenames; the tool description spells out both constraints and usage so the model writes compliant code on the first attempt. On failure, only the last 1500 characters of stderr are returned — error messages are tokens too.

flowchart TD
 A["agent calls make_figure {name, code}"] --> B{"Layer 1: ast_check static allowlist"}
 B -->|reject| E["Instructional error: allowed and banned items spelled out"]
 E --> F["agent rewrites code, retries"]
 F --> B
 B -->|pass| W["Wrapper: pre-import np/plt/loadmat, inject SRC/OUT/srcpath/listdir/walkfiles + _GUARD"]
 W --> R["Layer 2: isolated subprocess<br/>from-scratch env · isolated cwd · python -I · 90s timeout"]
 R -->|"OUT exists and non-empty"| P["PNG into session figure dir, embedded at compile time"]
 R -->|failure or timeout| E

Measured: seven classes of malicious samples, all blocked before execution

Verified by real runs inside the container. Two legitimate paths passed: basic matplotlib plotting, and plotting real data read via srcpath. Seven classes of malicious samples were all rejected by the AST layer before execution:

Malicious sample	Rule hit
`os.system` command execution	Dangerous-name blocklist (os is pre-imported; bare names are intercepted, not imports)
`os.remove` deleting files (no import — uses the pre-imported os)	Same as above
`import socket` for egress	Import allowlist
`eval` dynamic execution	Forbidden builtins
`__import__` bypassing the import statement	Forbidden builtins
Dunder escape (`__class__`/`__subclasses__`)	Dunder attribute interception
`open` reading /etc/passwd	Forbidden builtins

Layer 2 was never reached by this sample set — which is exactly as intended: a backstop’s value lies in unknown samples, not in the test set.

Two unexpected payoffs: the security constraint pays back at packaging time

For standalone distribution the backend is frozen into a binary with PyInstaller. Two interactions with the sandbox could have been deep pits; the allowlist made both shallow.

Payoff 1: a finite, known import set makes collect_all a precise fix. PyInstaller collects dependencies by static analysis, but the plotting code inside the sandbox is a string that exists only at runtime — whatever it imports is invisible to the packager. With an unbounded import set the problem is unsolvable; the AST allowlist, however, pins it down to a finite known set, so running collect_all over numpy/scipy/matplotlib and the rest captures everything. Measured on Windows: the frozen onedir output is ~260MB (30MB exe), with selftest reporting imports 10/10 and savefig 18250 bytes — the scientific stack is fully usable inside the frozen bundle.

Payoff 2: self-dispatch around the frozen sys.executable. In development the execution command is [sys.executable, "-I", script]; after freezing, sys.executable is this binary rather than python, and exe -I script would be interpreted as a server launch and fail. The fix: when run_plot detects is_compiled(), it instead sets the environment variable PDFAGENT_PYRUN=<script path> and launches another copy of the same exe; the entry point run_server._pyrun() sees the variable, executes the script via runpy.run_path, and exits without starting the server. Process isolation, the scrubbed environment, the timeout, the AST allowlist, and _GUARD all remain — only the -I flag is lost. The dispatch is pinned by the two cases in test_pyrun.py (with the variable set, the script runs; without it, no-op).

Where not to do this

This is not a full sandbox. There is no microVM and no container-level isolation; the subprocess shares the host’s kernel and filesystem permissions — a combined attack that bypasses the AST layer and evades _GUARD exists in theory. The project states its position in the module docstring: it targets the local single-user scenario (the code plots the user’s own local data — equivalent to the user running a script by hand), where two layers suffice; for untrusted folders or multi-user service, the real fix is one-shot, network-less container execution.
The allowlist sacrifices expressiveness. getattr and open are banned; h5py and similar libraries are not on the list, so HDF5-class data requires explicitly extending the allowlist — and re-assessing the new library’s capability surface (can it make network requests, can it write files). Every notch the allowlist widens, the weaker Layer 1’s guarantee becomes.
Domain restriction is the precondition. The approach works because the task domain is narrow: “data plotting” needs a small, stable module set. For general-purpose code execution — an agent freely writing arbitrary tool scripts — the allowlist would widen until meaningless; go straight to container/microVM isolation, with the AST layer at most as a heuristic pre-check.
Layer 2 couples to platform details. Building the environment from scratch means discovering every implicit platform dependency yourself: Windows system variables, matplotlib’s writable config directory. Projects distributing cross-platform should budget for verification by real runs, not by reasoning.

From chat to agent: tool loops and collapse breakers

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 4 of the Engineering LLM Applications series. The series is grounded in the implementation and measurements of a production-grade AI writing agent (FastAPI + React + Tauri, 351 test cases).

The problem: handing the loop to the model

The first three articles of this series were about single-call engineering: controlling input, assembling prompts, sanitizing output. The watershed of an agent is that what to do next is also handed to the model: give it a tool set, execute its calls, feed the results back into the message list, and call again — until the task is done.

The concrete scenario for this article: give the agent a data folder whose file names and contents are unknown in advance, and have it read files on its own, figure out what experiment this is, and write the report. The naive implementation is a while True tool loop, and in production it fails in three ways:

Untrusted paths. The model decides which file to read; a single ../ in an argument reads outside the data directory.
No natural endpoint. “How much exploration is enough” is the model’s judgment call; there is no upper bound on iterations, and therefore none on cost.
Loop collapse. The most insidious class: in long contexts, the model degenerates into calling the same tool with identical arguments. A public case, qwen-code issue #4695, records 43 consecutive git status calls consuming 8.9M tokens in one session; the deepseek-v4-pro tier we use is also prone to collapsing into this pattern in long contexts. SDK retries cannot help — retries only cover network errors and rate limits, while each of these calls “succeeds” at the API layer. The breaker has to live on the client side.

The rest of this article walks through five mechanisms in build order. The implementation uses standard OpenAI-compatible tool calling (DeepSeek by default; any compatible endpoint can be swapped in); the entry point run_agent is a generator that yields events one by one for SSE streaming.

Mechanism 1: a minimal tool set with a path guard

The exploration phase has five tools: list_dir, read_file, read_image (a local vision model extracts readings from screenshots and figures), make_figure (executes model-written matplotlib code to produce plots), and finish_exploration. Three design points:

Segmented continuation instead of stuffing everything in at once. read_file clips text at READ_CHARS = 7000 characters and, when it clips, appends one line: “truncated — use offset/limit to continue reading by lines”. Truncation is not an error; it is a decision point for the model: there is more, and whether it is worth reading further is the model’s trade-off to make. It also imposes a constraint on the loop-breaker fingerprint later — continuation reads with different offsets are legitimate behavior and must not be flagged.

Explicit phase transition. finish_exploration turns “exploration is done” from an implicit signal (the model simply stops calling tools) into an explicit call: the schema requires experiment / goal / key_findings, with three optional slots for assignment requirements, methodology notes, and a reference report. The phase transition leaves a structured record — and gives the breaker a well-defined exit to steer toward.

Every path entry goes through _safe.

def _safe(rel, root=None):
 root = root or base_docs_root()
 if not root:
 return None, None
 t = os.path.normpath(os.path.join(root, rel or ""))
 if not (t == root or t.startswith(root + os.sep)):
 return None, None
 return root, t

After normpath joining, the resolved path must still be inside the root; an escape returns (None, None), and the tool layer turns that into a “path does not exist or is out of bounds” string returned as the tool result — the model sees it and can correct itself, and the main flow never raises. The make_figure tool description also spells out hard sandbox rules (no importing os/sys/subprocess and the like, no open()/eval/exec, no double-underscore attributes); its two-layer sandbox implementation is the subject of the next article.

Mechanism 2: a zero-LLM scan before exploration starts

Before the model begins exploring, scan_folder performs a deterministic recursive scan: files are coarsely bucketed by extension into documents / images / data / other, returning the bucketed listing and counts — zero LLM calls, zero tokens. It serves two purposes: an instant whole-folder snapshot for the user (the detail of the first step event), and deterministic input for the later material-gap check. The principle is the same as in Part 11: “what is in the folder” is a question code can answer, and the model’s exploration budget should be spent on “which files to read and what to make of them”. One cost note in passing: the exploration tool loop runs on the cheap model tier (reading files and deciding what to read is simple work); drafting and reflection use the main model.

Mechanism 3: two lines of defense — a budget cap and a collapse breaker

MAX_TOOL_ITERS = 20 is a budget cap, not a defense: a collapsed agent will burn all 20 iterations on repeated calls and produce nothing. The real defense is _LoopBreaker, built around a call fingerprint:

def _tool_fingerprint(name, args):
 """Tool-call fingerprint = name + full arguments (sorted JSON)."""
 try:
 return name + "|" + json.dumps(args, ensure_ascii=False, sort_keys=True)
 except Exception:
 return name + "|" + str(args)

Three design constraints: full arguments — offset=0 and offset=100 are two different fingerprints, so legitimate segmented reads are never flagged (a unit test pins this property); sorted serialization — argument key order does not affect the verdict; never raises — serialization failure falls back to str(). Wrap-up tools like finish_exploration are excluded from counting.

The thresholds come in two tiers. Third occurrence of the same fingerprint (_LOOP_WARN): inject one wrap-up prompt — “You are repeatedly calling the same tool with the same arguments. What you have read is sufficient; call finish_exploration immediately and state your conclusions” — with nudge_once guaranteeing a single injection per run, because the injection itself occupies context and repeated injections would accelerate the collapse. Fifth occurrence (_LOOP_STOP): hard-break out of the exploration loop. The break is not a crash: the agent yields a “repeated identical tool call detected — exploration cut off, writing from what has been read” event and proceeds to the drafting phase with the material accumulated so far. A degraded deliverable plus a human reviewer beats zero output after the budget is exhausted.

flowchart TD
 A[Call model with TOOLS] --> B{tool_calls returned?}
 B -->|none| N[Inject: keep exploring or call finish_exploration] --> A
 B -->|finish_exploration| W[Record understanding, enter drafting]
 B -->|other tools| F{LoopBreaker fingerprint count}
 F -->|ok| E[Execute tool, feed result back] --> A
 F -->|3rd time, once only| G[Execute tool, inject wrap-up prompt] --> A
 F -->|5th time| X[Hard break after this round] --> W

The whole loop still sits inside for _ in range(MAX_TOOL_ITERS) — the two defenses hold independently. The breaker has an end-to-end test: a mock client that, whenever tools are passed, always returns the same read_file(a.txt) call (simulating total collapse); the test asserts that the break event must appear and that the number of create calls is ≤ 6 — far below the cap of 20. The threshold sequence (first two ok, third nudge, fifth stop) and the fingerprint properties are pinned by separate unit tests.

Mechanism 4: the inverse failure in drafting — tool markup without tools

After exploration, drafting begins, and this phase passes no tools. But the model may still emit its internal tool-call markup (DeepSeek’s DSML) as body text — the generation “succeeds” on the surface, and the resulting report is garbage. _complete_report validates output with three deterministic criteria: contains \documentclass, contains no DSML, contains no tool_calls. On failure it injects a reinforcement instruction (“You have no tools available right now… output the complete LaTeX source directly”) and retries, up to 3 times. This is the output-sanitizing approach of Part 3 extended into the agent context: a structurally invalid output is not patched textually — the constraints are tightened and the call is retried.

Mechanism 5: deterministic progress — pre-enumerated milestones and explicit stages

A streaming agent’s progress bar cannot run on guesswork. This pipeline is “process as configuration”: scan → explore → [figures] → [gap] → draft → reflect×N → bib → [compile], with the optional steps switched by the template. So _plan_pipeline enumerates the milestone sequence before the run starts, from configuration alone; M = len(plan), every milestone event carries index/total, and the progress bar has a fixed denominator from the first second. Two consistency details:

The gates for optional steps must match the runtime decisions verbatim: has_xelatex is probed exactly once, and the plan and the runtime compile gate reuse the same result, so the two can never diverge;
the stage name is passed explicitly rather than looked up from _plan[_pi-1]: the reflection loop may break early on convergence, and a skipped milestone would make the positional lookup mislabel the subsequent bib / compile milestones as reflect.

One known small deviation is documented honestly in the code comments: the gap step also depends at runtime on whether material exists and may actually be skipped — the plan counts it anyway, and the completion event backfills to 100%.

Measurements

Real-run validation (devlog record, DeepSeek v4-pro, an electromagnetics field-scan dataset): the agent ran list_dir automatically and made multiple rounds of read_file (a PDF, band.csv, scan parameter files, and more), and its autonomous identification of the experiment was correct; the draft was 16,773 characters, and after one reflection-revision round the final report was 20,742 characters across 9 sections, with the full run taking 745 seconds. That validation ran on the initial version whose tool set was only list_dir / read_file / finish_exploration; read_image and make_figure were added later.

Mechanism	Alternative	Effect
`_safe` path guard	Trust model-supplied paths	Escapes become tool-level errors the model can self-correct; nothing raises
`scan_folder` pre-scan	Let the model map the folder itself	Zero-token bucketed snapshot of every file
`MAX_TOOL_ITERS = 20`	Unbounded loop	Hard cost ceiling
`_LoopBreaker`	Iteration cap / SDK retries alone	≤ 6 create calls in the collapse scenario (end-to-end assertion), saving 14+ wasted rounds
DSML detection and retry	Accept the “successful” output	Valid LaTeX within at most 3 retries
`_plan_pipeline` + explicit stage	Guess progress from event counts	Denominator known before the run; early convergence never mislabels stages

Where not to do this

An exact fingerprint catches total collapse, not wandering. Low-value calls to the same tool with varying arguments (reading irrelevant files one after another) never trip the breaker; that class is bounded by the budget cap and prompt quality. Fuzzing the fingerprint (ignoring some arguments) would widen coverage but misfire on legitimate continuation reads — we chose narrow and precise.
The nudge works only while the model still follows instructions. The wrap-up prompt helps with mild circling; once the context has collapsed severely, the model no longer responds to injections and only the hard break matters — which is exactly the rationale for the two-tier thresholds.
A degraded deliverable requires a human in the loop. A report written from partial material is only worth something if someone reviews it downstream. If the agent’s output is executed automatically and partial information can cause harm, the breaker policy should be fail-the-whole-run rather than degrade.
Pre-enumerated progress requires a fixed pipeline. _plan_pipeline works because the pipeline structure is settled before the run starts; an open-ended agent whose plan is generated dynamically by the model has no a-priori milestone sequence, and progress degrades to event counting.
The thresholds are not universal constants. 20 / 3 / 5 were tuned to the scale of “read a folder, write a report” and have not been validated on other task shapes.

References

qwen-code issue #4695 — a public case of tool-loop collapse (43 repeated git status calls / 8.9M tokens in one session).

Making model output usable: sanitizing and automatic repair

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 3 of the Engineering LLM Applications series. The evidence base is a production-grade AI writing agent that generates LaTeX lab reports; every error message and number in this article comes from its source code and real debugging records.

The problem: the distance between “looks correct” and “usable”

You ask the model to generate a LaTeX lab report. The source it returns is structurally complete, every command spelled right, nothing visibly wrong; you feed it to xelatex and get no legal \end found — even though \end{document} is plainly there at the end of the file. The real cause hides in the header: the model wrote \def\partnerID{% TODO...}, and the % inside the \def line comments out the closing }. A runaway definition swallows everything after \begin{document}. The error location and the cause location are an entire document apart.

This is the most typical gap you hit when starting out with LLM applications: model output and usable artifact are not the same thing. Taking “generate LaTeX → produce a PDF” as the running example, at least three failure layers sit in between:

Won’t compile: syntax looks fine, but it trips corner-case behavior of the engine-and-package combination (the % above; \mathbf below);
Compiles but unusable: compilation “succeeds”, yet the artifact has 0 pages, or figures are missing and every reference renders as [?];
Usable but unsafe: under -shell-escape, one \write18 in the generated body is arbitrary command execution.

The common first reaction is to tighten the prompt or retry the model. Our measured conclusion: prompt constraints lower the incidence but cannot suppress it (the verbatim source comment reads “the model routinely omits these; the prompt can’t hold it down”); retries cost another generation and don’t guarantee convergence. Another counter-intuitive result came from root-causing three classes of generation-quality problems: 2 of the 3 root causes were in our own backend code and template, not in the model. The right place for the fix is a layer of deterministic sanitizing and repair between model output and artifact — zero tokens, unit-testable, predictable behavior. It is also the output-side concretization of the deterministic boundary principle from Part 11 of this series: no fragment enters system state without passing a deterministic gate.

Entry cleanup: strip_fragment and check_fragment

The first gate sits before a fragment enters the document, and handles domain-independent shape problems:

def strip_fragment(text: str) -> str:
 """Clean a model-returned section fragment: strip Markdown code fences;
 if a full document came back by mistake, cut from the first heading to just
 before \end{document} (keeps preamble / top-level end out of the body)."""

Models love wrapping code in Markdown fences; asked to “return only this section”, they may still disobey and return the whole document. The former gets unwrapped; the latter is cut from the first \section-family command to just before \end{document}. The companion check_fragment runs a health check: balanced braces, balanced \begin/\end, no \documentclass, non-empty — all soft signals that warn rather than block, because the check rules themselves can produce false positives.

The pit catalog: deterministic rewriting in sanitize_tex

The second gate runs before compilation and is domain-specific: a pit catalog, every entry earned through a real debugging chain — fixing one exposed the next:

Symptom (real error)	Root cause	Deterministic fix
`no legal \end found` (with `\end{document}` present)	`{%` inside a `\def` line comments out the closing `}`; runaway definition swallows the rest	`{%`→`{}%` on `\def` lines only; the multi-line `\abstract{%` idiom is left alone
`\textfont 11/12 undefined` (`\mathbf`); `Extended mathchar used as mathchar` (`\boldsymbol`)	both conflict with the xelatex + unicode-math combination	rewrite uniformly to unicode-math’s `\symbf`
`Extended mathchar` / `A number should have been here`	bold inside accents (`\hat{\boldsymbol{x}}`)	drop the bold, keep the accent; both nesting orders handled
A string of Undefined errors	template commands like `\makeEngPage` called with their prerequisite variables unset	inject empty defaults after `\begin{document}`
A missing-`\item` / `\noalign` cascade	the two-column table environment instrmlist used as an itemize with `\item`	rewrite `\item Name (Model)` inside the block to `Name & Model \\`; already-correct rows untouched

Two entries deserve expansion.

We fixed the \mathbf entry in the wrong direction once. The first fix was \mathbf→\boldsymbol (the template loads the bm package — seemingly reasonable). Only an end-to-end rerun revealed the direction was wrong: \boldsymbol triggers the same “Extended mathchar used as mathchar” under xelatex + unicode-math. The correct target is unicode-math’s own \symbf. After the correction, recompiling a historical version containing 13 occurrences of \mathbf measured: Extended mathchar 13→0, total errors 48→35. The lesson is twofold: sanitizing rules need empirical verification of their own, or the fix itself introduces regressions; and the rewrite target must be a form that is provably compatible with the current engine combination, not one that is “usually equivalent”.

What value to backfill is a product judgment, not just a technical one. The template-variable backfill carries an exception table: when the English-page author name is missing, the value of the corresponding Chinese variable is copied — author names are shared across both language pages, and that beats an author line reduced to a lone “and”. But the English abstract abstractEng is deliberately left empty: an abstract requires translation, and pushing Chinese text onto the English page would inject wrong information. The red line for automatic repair: backfill only defaults that are provably harmless; never guess semantics.

Artifact and safety: figure placeholders, font substitution, injection stripping

Some problems live not at the syntax layer but at whether the artifact is usable and safe to distribute.

A missing figure takes down the whole document. Models routinely \includegraphics figures that don’t exist in the provided material. A missing figure doesn’t just report File not found — it cascades into a truncated .aux: the whole document renders 0 pages and the PDF won’t open. The fix is a deterministic substitution: any \includegraphics referencing a file absent from the figure directory is replaced with an \fbox placeholder stating “this figure was not provided — supply it or remove the reference”. Zero LLM calls, two error classes eliminated at once, and a readable multi-page PDF comes out.

Platform substitution for a hard-coded font. The template’s cls hard-codes \setmainfont{Times New Roman} (a Windows-bundled font); a Linux container doesn’t have it, producing 100+ font errors per report. The compile step edits only the working-directory copy: on non-Windows platforms the font is replaced with the always-available Latin Modern Roman; the template itself is untouched. The dev machine (WSL2) taught a related lesson: Liberation’s font aliases do not satisfy XeTeX/fontspec’s exact-name matching — installing the real MS fonts took font errors from 6 to 0.

Injection stripping. The template’s minted 2.x forces full -shell-escape (minted.sty:1233 checks \pdf@shellescape=1 and refuses restricted mode), and -shell-escape lets LaTeX run arbitrary shell commands — which makes LLM-generated body text a genuine attack surface. The sanitizing layer strips command-execution primitives from the body: \write18, \ShellEscape, \directlua, and the piped form \input{|cmd}. minted’s own pygmentize invocation lives in the package layer, not the body, so syntax highlighting is unaffected. End-to-end verification: after injecting \immediate\write18{touch ...} and compiling, the target file was not created. The standalone distribution drops -shell-escape entirely — code blocks still typeset via listings, losing only syntax coloring.

After compilation: diagnose and humanize

“Compilation succeeded” lies: under nonstopmode, missing figures and undefined references don’t stop a PDF from being produced. diagnose deterministically tallies four items from the log — error-line count (^!), the list of missing figures, undefined citations, undefined cross-references — so that “succeeded” no longer masks a degraded artifact.

The tally serves two audiences. For the model, extract_errors picks the !-prefixed error lines plus context as input to the compile self-repair loop (Part 6). For the user, humanize maps diagnostics and common errors into “one plain sentence + one clickable action” — along the lines of “2 figures missing: a.png, b.png” with a button “supply the figures, or remove the corresponding \includegraphics”; “Undefined control sequence” becomes “an undefined command was used (possibly a typo or a missing package)”. The raw log is for engineers; it has no place in the product UI.

The pipeline, and the measurements

flowchart TD
 A[Model output] --> B[strip_fragment: unfence / cut fragment from full doc]
 B --> C[check_fragment: balance checks, soft signals]
 C --> D[sanitize_tex: pit-catalog rewrites + exec-primitive stripping]
 D --> E[Assemble workdir: placeholder missing figures; swap font off Windows]
 E --> F[xelatex + bibtex]
 F --> G[diagnose: error count / missing figures / undefined refs]
 G -->|for the user| H[humanize: one sentence + one clickable action]
 G -->|for the model| I[extract_errors → compile self-repair loop]

Measured item	Before	After
`\mathbf`→`\symbf` (recompiling a version with `\mathbf`×13)	Extended mathchar 13, errors 48	0, errors 35
Photonic-crystal auto report (after the first three fixes: `%` in `\def`, math bold, bold-in-accent)	compile failure (runaway)	valid 131KB PDF
Real MS fonts installed on WSL2 (aliases fail exact-name matching)	6 font errors	0
Injected `\immediate\write18{touch ...}`	—	file not created (RCE blocked)

The sanitizing layer is pure functions throughout; when the \symbf fix landed, the full pytest run was 37 passed — every rewrite rule, every backfill exception, and the injection stripping is pinned by an assertion.

Where not to do this

The pit catalog only fixes known pits. Every rule is bound to a specific combination (GPE template + xelatex + unicode-math + minted 2.x); a new template or engine means walking the debugging chain again. It is not a general-purpose LaTeX fixer, and should not try to become one.
Regex rewriting has an inherent collateral-damage surface. Every rule must be narrowed to the lesion: only \def lines, only inside the instrmlist block, already-correct rows preserved verbatim — and pinned by unit tests. One notch too wide and the rule breaks legitimate input; anything that cannot be narrowed to provably-safe should not be auto-fixed, and belongs to the model-driven compile self-repair loop instead.
The ceiling of automatic repair is introducing no wrong information. Deterministically decidable defaults (empty variables, placeholder boxes) may be backfilled; semantic gaps (the English abstract) must be left to a human or the model. Cross that line and repair becomes contamination.
Sanitizing replaces neither upstream constraints nor the downstream loop. The “LaTeX compilability rules” still belong in the system prompt (they lower incidence), and compile self-repair handles the long tail; the sanitizing layer’s role is zero-token interception of high-frequency known pits. The three layers are a union, not substitutes.
Injection stripping is one layer of defense in depth, not all of it. The regex layer does not cover extreme bypasses such as catcode tricks — an acceptable trade-off for a local single-user tool; a multi-tenant server must stack harder layers (a restricted shell-escape baseline, sandboxed execution environments).

Multi-agent adversarial review as an engineering practice

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 13 of the Engineering LLM Applications series.

The problem: the typical failure of an LLM finding is that it looks real

When you use an LLM to review code, volume is never the bottleneck — it will happily list fifty “potential issues” against any codebase. The bottleneck is at the other end: the typical failure mode of an LLM finding is plausible-but-wrong. The line number is specific, the failure narrative is complete, the suggested fix is fully formed — and then it falls apart the moment you confront it with the code: the guard already exists, the alleged race path is unreachable, the “leaked” resource is reclaimed somewhere else.

A review pipeline without a verification stage drowns in fake findings, and fake findings cost you both ways: either you falsify each one by hand, paying back every hour the review saved, or you start ignoring the report wholesale — and the real problems die with the fake ones. The second outcome is the more common and the more fatal: a report that mixes true and false findings carries close to zero engineering information.

A separate, independent problem is breadth: a single-perspective review — whether a human or a single agent — cannot stay simultaneously alert across security, concurrency, billing, and cross-platform consistency.

Both problems point at the same structural remedy: split “find problems” and “verify problems” into two stages with opposite objectives — the finding side optimizes for coverage, the verifying side optimizes for truth. This article walks through the structure, grounded in two production runs (one full-project audit, one session-level review) plus one set of overturn data.

The structure: parallel discovery by dimension, adversarial verification per finding

Stage one: N auditors in parallel, each guarding exactly one dimension. The full-project audit used 6 (backend core, API-DB-security, LaTeX-verify-distill, frontend, test blind spots, documentation drift); the session-level review used 4 (concurrency/runs, security, frontend regression, data integrity). The point of slicing by dimension is not parallel speed-up but forced perspective: an auditor that only looks at concurrency does not have its attention taxed by functional correctness.

Findings obey a format discipline: every one must land as file:line + a concrete failure scenario + a proposed fix. This is the precondition for everything downstream — it turns a finding from “something feels off” into a falsifiable claim.

Stage two: each finding goes to an independent verifier. The verifier is not the agent that produced the finding and does not enter with that agent’s reasoning in context; its instructions are default skepticism, specialize in falsification — go into the code and look for evidence that the finding does not hold. The verdict is three-valued: CONFIRMED / uncertain / dismissed, and only CONFIRMED enters the repair queue.

Independence is a requirement, not an option. When one agent checks its own work, its finding is sunk cost sitting in its context, and it will defend it; only when “successfully overturning the finding” is the verifier’s success criterion do the incentives align. This is the same issue as judge validity from part 9 of this series, seen from the other side: the verifier is itself a judge, and its validity has to be demonstrated by the fact that it actually overturns things — data below.

flowchart TD
 A[Codebase / change set] --> B1[Auditor: security]
 A --> B2[Auditor: concurrency]
 A --> B3[Auditor: ... N dimensions total]
 B1 --> F[Finding: file:line + failure scenario + fix]
 B2 --> F
 B3 --> F
 F --> V{Independent verifier: default skepticism, falsify}
 V -->|CONFIRMED| T[Triage by severity]
 V -->|dismissed| X[Discard]
 V -->|uncertain| U[File as open question]
 T --> R[Fix, or defer explicitly with a recorded reason]
 R --> P[Regression tests confirm no breakage]

Two measured runs

	Full-project audit	Session-level review
Scope	Entire codebase (backend + frontend + docs + tests), focused on the broader/older parts	One session’s changes: 44 files / 1,467 lines, including a core-path refactor
Auditors	6, by subsystem/dimension	4, by risk dimension
CONFIRMED	27 (5 high / 14 medium / 8 low)	7
uncertain / dismissed	0 / 2	all others falsified
Disposition	24 fixed, 3 explicitly deferred	7 fixed
Post-fix verification	pytest 243 passed (+6 new tests), vite build clean	pytest 18 passed + targeted live tests

Beyond the counts, what deserves attention is the type of problem on the confirmed list — a substantial share is the kind a single person reading the diff end-to-end would very likely miss:

TOCTOU race in the 409 concurrency guard (session-level): “check whether a run is already running” and “create the run” were not atomic, so concurrent requests could double-start. The fix moved the check, the creation, and the handle registration together inside runs._LOCK; the repair was closed out empirically — two concurrent requests return [200, 409].
Chat billing used the unresolved provider (session-level): with DEFAULT_PROVIDER=anthropic and the request omitting the provider field, cost was computed at deepseek’s unit prices — roughly a 60× underestimate. Provider resolution lives in one place, pricing in another; reviewed module by module, both sides “look correct”.
Anthropic streaming max_tokens=32000 while DeepSeek’s was 64000 (full audit, medium): long reports silently truncated on one provider; only a cross-provider consistency comparison exposes it. Fixed by unifying on _STREAM_MAX_TOKENS=64000.
Figure sandbox environment hardcoded POSIX (full audit, high): on the Windows distribution (the primary platform), the CPython subprocess cannot start for lack of SystemRoot/TEMP, so make_figure crashes unconditionally.
EXPOSE_RESET_TOKEN defaulting to true (full audit, high): any anonymous caller could obtain a reset token and change a password — account takeover; changed to a secure default of false.

These respectively require concurrency-interleaving reasoning, cross-module dataflow tracing, cross-provider comparison, and cross-platform runtime knowledge — exactly the categories that one dedicated perspective per dimension covers and one sequential read-through misses.

One disposition detail worth copying: the 3 deferrals are not silent omissions but explicit decisions with recorded reasons — all are SaaS-only issues neutralized in the shipped STANDALONE configuration by _no_cloud/forced-local, and fixing them outright would have meant disturbing a set of non-STANDALONE test semantics; they are filed for whenever SaaS is revived. The output of a review is not just fixes; it is also this traceable record of why not.

Verification is not a rubber stamp: overturn data

Whether the adversarial verification stage actually works needs evidence of its own. The same find-then-adversarially-recheck structure was applied to a different kind of object: auditing whether the writing rules produced by the distillation experiment (part 12 of this series) genuinely learned “judgment” rather than reskinning structural templates. Eleven agents in total: for each rule pair, an item-by-item initial classification, followed by an adversarial re-review whose instructions were, again, default skepticism — specializing in items whose “because” is a tautology or whose structural action carries only a thin veneer of justification.

Pair	Initial judgment share	After adversarial re-review	Pair verdict
1	87%	80%	holds
2	82%	73%	holds
3	73%	67%	holds
4	82%	64%	holds
5	100%	90%	holds

The initial numbers of all five pairs were revised downward; the post-adversarial mean is 0.746, and the overturned items follow one highly consistent pattern — the action overlaps a structural template and the stated reason is close to a tautology. The pair-level verdicts held 5/5, but no pair emerged untouched. The original report also flags, unprompted, that pair 5’s 0.9 is optimistic (the initial 1.0 was itself suspicious).

This data establishes two things. First, the verifier really does overturn items case by case — it is not process decoration. Second, the value of the adversarial stance runs in both directions: in code review it kills false positives (2 dismissed; “all others falsified” at the session level), and in evaluation tasks it deflates optimistic self-assessment (100% → 90%).

Cost and applicability

The cost structure is straightforward: every finding pays for one additional dedicated verification pass, plus orchestration overhead. When it is worth it:

Large change surfaces. The session-level run covered 44 files / 1,467 lines including a core-path refactor (“merging chat into run”) — a surface beyond what one person can hold in a single pass.
Pre-release / security surfaces. Problems like EXPOSE_RESET_TOKEN or the Stripe webhook skipping validation when no secret is configured (fixed to fail closed: 503 without a secret) have a miss cost wildly out of proportion to the verification cost.
Legacy code never systematically audited. The full-project audit’s stated goal was precisely “not just the new code — focus on the broader/older parts”, code outside the field of view of any recent review.

When it is not worth it, the source record already contains the judgment: single-feature implementation is “sequential and interleaved, with no fan-out” — solo work suffices; review and audit (breadth plus independent verification) are what the multi-agent workflow is actually good at. Extrapolating to small changes: on a few dozen lines of diff, falsifying three findings by hand is far faster than orchestrating a six-auditor pipeline, and the verification overhead simply exceeds the benefit.

Limits

The shared-origin blind spot. The verifier and the author of the code under audit are the same (or a sibling) LLM. The adversarial structure removes positional bias — no one defends their own output anymore — but it does not extend the capability boundary: a category of problem invisible to both sides stays invisible to the verification stage too. It is no substitute for signals of a different origin — actually running the code, live concurrency tests, human domain knowledge.
“0 uncertain” does not mean “0 escaped”. 27 CONFIRMED with 0 uncertain measures the precision side: nearly everything raised was real. It says nothing about the recall side — the problems no auditor raised at all. This very audit confirmed three test blind spots: an audit can find the holes in the tests, but there is no instrument measuring the holes in the audit.
Verification depth is itself sampled. The overturn experiment’s report states honestly that only pair 4’s source file was empirically spot-checked; the rest relies on sub-agent labeling. Verification can be shallow, and a CONFIRMED stamped by shallow verification deserves a discount — mistaking “verification happened” for “verification was thorough” is this pipeline’s most seductive self-deception.
The structure depends on finding discipline. file:line + failure scenario + fix is what makes a finding a falsifiable claim; relax it to “module X may have issues” and the verifier has nothing to confront — the whole pipeline degenerates into two rounds of opinion exchange.

Prompt assembly as engineering: layering, contamination, and trimming

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 2 of the Engineering LLM Applications series, grounded — like the rest of the series — in the real source code and tests of a production-grade AI writing agent.

The problem: when the prompt grows too long for anyone to dare touch

When your first LLM feature ships, the prompt is usually one f-string: a role blurb, format requirements, a few examples, the user input, concatenated and sent. That stage is fine — the problems arrive three months later:

Fix one place, break another. The same prompt text is reused by several call paths; wording tuned for scenario A quietly degrades scenario B’s output, and no test turns red.
Example content leaks into output. To teach the model how to write, you paste in a model essay; one day a user finds that essay’s numbers and citations inside their own report.
Multi-turn gets more expensive every turn. Each turn carries the full history, and the history contains entire documents the model previously produced; token spend climbs with turn count, most of it redundant.
Changing a sentence requires a release. The prompt is hard-coded in source; tweaking one line of wording in the field means shipping a new package.

These four failure classes share one root cause: the prompt is treated as a piece of text rather than a build artifact. A build artifact implies three things: a layered structure (each layer with its own change frequency and ownership), incidents (which need incident records), and regression protection (changes are guarded by tests). The rest of this article dissects that practice through prompts.py and its tests, from this series’ evidence-base project (an AI writing agent producing LaTeX lab reports).

Mechanism 1: layered assembly — each layer has its own change frequency

The system prompt is a concatenation of five layers (build_system), not one string:

Layer	Content	Change frequency	Hot-swappable
Role boundary	Identity + hard prohibitions (no AI-speak, no fabricated data or citations)	Very low	Via overlay
Genre spec	Two section skeletons: lab report / course notes	Low	Via overlay
Quality anchors	Writing methodology + golden sample (permanent domain assets)	Low	Via overlay
Writing-judgment skill	Distilled “why we write it this way”	High	Distillable, replaceable
Output-format hard rules	LaTeX template skeleton + 6 compile-error pitfalls	Tracks the template	Via overlay

Three design points:

Move knowledge assets out of code. The methodology (~4KB of Markdown) and the golden sample (a 2.2KB LaTeX technique excerpt) live under the knowledge/ directory as file assets; code only assembles. Together with the bib file and the material pack they form the agent’s domain assets — reviewable and replaceable independently of the code.

Write format rules with their error messages attached. The output-format layer does not just say “do this”; it lists six numbered pitfalls, each with the real compile error that violation triggers — e.g. “before calling \makeEngPage{} you must \def the English metadata … missing any one of them raises Undefined control sequence”, and “bold vectors only with \symbf{}; \mathbf triggers Extended mathchar”. These were baked into the prompt after real failures — the compiler’s error-driven experience, front-loaded.

Trim by call site. Section-level small edits (scoped editing, see The deterministic boundary) assemble with scoped=True: the quality anchors are dropped — methodology plus golden sample come to roughly 3.2k characters, about 2k tokens, and a one-section fragment edit has no use for a full model essay about someone else’s experiment. The role boundary, the distilled judgment layer, and the format rules stay, because the fragment must still be stylistically correct and compilable.

flowchart TD
 A[Assemble system] --> B[Role boundary]
 B --> C[Genre spec: report or notes]
 C --> D{Scoped small edit?}
 D -->|yes| E[Skip quality anchors, keep distilled judgment layer]
 D -->|no, full generation| F{Distilled skill present?}
 F -->|yes| G[Fixed quality anchors, dedup, then judgment layer]
 F -->|no| H[Built-in judgment layer = the anchors]
 E --> I[Output-format hard rules]
 G --> I
 H --> I

Mechanism 2: a real incident — the quality anchors got skipped

Layering is not about tidiness on paper; it exists because not layering caused a real incident. The incident record sits in the docstring of build_system itself (translated; the source is Chinese):

"""⚠️ An early L1 refactor made the "distilled template" path (skill_prompt
non-empty) skip builtin_skill entirely → golden sample + methodology lost →
generation depth/style collapse (the root cause of "output is worse with
distilled templates"). Quality anchors are now extracted and always injected,
with dedup for old built-in/legacy templates that carry their own anchors."""

The post-mortem: in the early implementation the quality anchors were bundled inside the “built-in skill”; a refactor made the “use a distilled template” path skip the built-in skill entirely — golden sample and methodology vanished together, and generation depth and style collapsed. The symptom read as “distilled templates produce worse output” and was for a while investigated as a distillation-quality problem; the actual cause was a missing layer in assembly.

The fix hardened into two structural decisions. First, decouple quality anchors from the skill layer: the anchors are “permanent domain assets”, independent of whether a distilled template is in use; a distilled template carries only the judgment layer (why to write it this way) and is not allowed to displace the depth/style anchors. Second, dedup for legacy templates — old built-in templates that already carry the anchors are not injected twice:

if skill:
 anchors = _quality_anchors(kind)
 if anchors and "金样例" not in skill: # skip if the skill already embeds the golden sample
 parts.append(anchors)
 parts.append(skill)

The transferable lesson: layered assembly must state explicitly which layers are unconditionally present — otherwise any “looks equivalent” refactor can silently remove a layer. A prompt missing a layer throws no error; it only degrades.

Mechanism 3: few-shot contamination guards

The golden sample is a double-edged sword. It is a model-essay excerpt about YBCO — someone else’s experiment — used to teach “how to write with depth, to spec, and compilably”. The natural risk of few-shot is content contamination: the model copies the sample’s subject, numbers, and citations into the user’s report — fabricating data by way of the example. There are three guards, all at the wording level:

An isolation label at injection time. The assembly code wraps the golden sample in a declaration: “it is a model essay about another experiment — never copy its subject / title / keywords / numbers / citations / section layout; you must write strictly from this session’s materials, about the experiment you actually did.”
The sample file carries its own immunization comments. fewshot_ybco.tex is not a complete report but a 2.2KB technique excerpt: only an equation-bearing principle paragraph, a booktabs table, and similar craft demonstrations survive. Its header comment restates “the topic below is only an example … never copy this excerpt’s title, data, citations, or sections”, and even the table caption reads “example — a real report must contain this experiment’s real data”.
The methodology layer adds an iron rule. The methodology file’s own hard-rules section independently repeats: “never copy the model essay’s / golden sample’s subject, numbers, or citations — that is someone else’s experiment.”

The triple redundancy is deliberate: wording-level protection is probabilistic, and a single declaration can be diluted in a long context. The final backstop is not in the prompt layer at all — a provenance check runs server-side against the full materials, deterministically. That belongs to the deterministic-boundary side of the system.

Mechanism 4: multi-turn history folding and material trimming

In multi-turn revision, what the model needs is the thread of per-turn feedback (what the user objected to first, what changed next) — not the full text of every historical version. Yet the assistant’s history messages are precisely entire LaTeX documents. The folding rule in build_messages:

if role == "assistant" and "\\documentclass" in content:
 content = "[此前已生成的文档版本，此处省略源码]" # "earlier document version, source elided"
elif len(content) > max_chars:
 content = content[:max_chars] + "…（略）"

Three points. Full documents are replaced by a one-line placeholder — the current document is already supplied separately in this turn’s user content, so re-sending every historical version burns tokens for zero value. Non-document assistant notes are kept — e.g. an auto-exploration conclusion like “the data is in the xrd/ directory” is a fact later turns need. Everything else over 4,000 characters is truncated, and only the last 40 history messages are kept. Dedicated tests pin this behavior (document folding, note preservation, count capping).

Materials get the same deterministic trimming (zero LLM calls, zero tokens): in revision turns, materials over 12k characters are filtered by lexical overlap with the current feedback; pure-number matrix lines (bare CSV) are always dropped — the figure pipeline has already digested the raw data, and prose writing has no use for row-level numbers. If filtering keeps fewer than two lines and under 40 characters, it falls back to the head of the material — filtering must never destroy all context. On the chat path, materials become a per-file-quota stable digest (each file guaranteed at least a 600-character allowance, so later files are not starved by earlier large ones), depending only on the material itself and therefore byte-identical across turns — usable as a stable first-message prefix that hits DeepSeek’s automatic prefix caching.

Mechanism 5: the overlay hot-swap layer and its regression gates

The last cost of hard-coded prompts is release coupling: changing one sentence in the field means shipping a new installer. The remedy is the agent-pack overlay:

def _prompt_part(rel: str, default: str) -> str:
 return _overlay_read(rel) or default

Role, genre specs, format rules, methodology, and golden sample are all overridable: a same-named non-empty file in the overlay directory wins; otherwise the built-in default applies. _overlay_read reads on every call (lazy loading) — swap the pack and the next generation uses it, with no recompile and no restart; a failed read or an empty file returns the empty string and falls back to the built-in, so an empty pack can never wipe out the knowledge.

The danger of a hot-swap layer is that behavior with no overlay must not change by a single byte. Two groups of offline tests guard this:

Overlay behavior tests: no overlay → bundled defaults; overlay takes precedence and is lazy (change the content from V1 to V2 and the next call sees V2); an empty file falls back to the built-in.
Prompt-shape assertions in the golden regression gate: the scoped prompt must still contain the “rewrite only this block” instruction and the prohibition wording (with \documentclass and \end{document} appearing inside the prohibition); build_user(gaps=None) must be byte-identical to calling without gaps — proving a newly added parameter did not contaminate the existing path.

Both groups are plain pytest, fully offline: whoever edits the prompt back to “output the full document”, or lets the default path drift, turns CI red immediately.

Costs and gains

Mechanism	Cost of not doing it	Effect in practice
Layering + scoped trimming	Full system prompt on every small edit	~2k tokens of quality anchors saved per scoped edit; with scoped editing, output tokens drop an order of magnitude
Fixed anchor injection	Relying on each template to carry its own	Fixed the “distilled-template collapse” incident; dedup prevents double injection
Contamination labels × 3	Bare few-shot, model-essay content leaking	Three wording-level guards + server-side provenance backstop
History folding + material trimming	Re-sending all history and materials every turn	Full documents fold to a one-line placeholder; 40-message / 4,000-character double cap; stable prefix hits caching
Overlay hot-swap	Every wording change is a release	Swap the pack, next generation applies it; no-overlay behavior byte-identical, guarded by golden gates

Where not to do this

Do not layer during prototyping. With one call path and a prompt still changing daily, one f-string is the correct form; layering and golden gates are for the stage where the prompt has stabilized, multiple call sites reuse it, and the field needs hot swaps. Premature abstraction pays every cost twice.
Shape assertions do not test semantics. The golden gate asserts keyword presence and byte identity; a rewording that preserves the keywords passes green while quality shifts. Semantic drift needs online evaluation sets — see part 8 of this series, The testing pyramid for LLM applications.
Contamination guards are probabilistic. Isolation labels lower the copy probability; they do not zero it. High-risk domains (numbers, citations) need deterministic verification outside the prompt as the backstop — the defense cannot live in wording alone.
Hot swapping cuts both ways. The overlay lets a field pack bypass the bundled tests — golden gates guard the built-in defaults, not the pack’s contents. Hot-swap capability must come with an acceptance process for the pack itself; otherwise it is an unguarded change channel.

The deterministic boundary: a first principle for LLM application architecture

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 11 of the Engineering LLM Applications series — where the mechanisms of the preceding ten articles converge into one principle. The series is grounded in the implementation and measurements of a production-grade AI writing agent (FastAPI + React + Tauri, 351 test cases).

The problem: probabilistic components, engineering obligations

Once an LLM is embedded in a production application, the hard engineering problem is not that the model is insufficiently smart. It is that a probabilistic component degrades system properties in several predictable ways:

Scope escape. Ask the model to “fix one section” and it rewrites the whole document. Empirical measurement of full-rewrite revision (arXiv:2601.13217) shows revisions regress 16–27% of already-covered content on average — not a model slip, but a statistical property of open-ended rewriting.
Loop collapse. In long contexts, agents degenerate into calling the same tool with identical arguments. A public case (qwen-code issue #4695) records 43 consecutive git status calls consuming 8.9M tokens in one session; SDK-level retries cannot help, because every call succeeds at the API layer.
Cost drift. If every turn carries the full context, a “fix the keywords” request pays for the entire document plus all source material — we measured roughly 6–7k input tokens for such requests, of which about 1.5k carries information.
Untestable regressions. Non-deterministic output defeats conventional assertions; a prompt edit silently shifts quality and CI has nothing to catch it with.

Against these failure modes, our project converged on one architectural principle, which this article calls the deterministic boundary:

Never ask the model a question deterministic code can answer; let the model produce only the minimal necessary fragment; admit no fragment into system state without passing a deterministic gate.

The principle is orthogonal to two existing families of tooling: structured outputs / function calling address format determinism (what the output looks like); guardrail frameworks address content compliance (what the output may say). The deterministic boundary addresses scope and state determinism — which part of the system the output is allowed to modify, and under what conditions it lands. The str_replace / fast-apply pattern in code editors (model produces the diff, application is deterministic) is the same principle expressed in another domain.

The rest of this article walks through five mechanisms, using document revision as the running example. The stack is Python/FastAPI and the document format is LaTeX, but the mechanisms are format-agnostic: they apply to any domain with parseable structure.

Mechanism 1: deterministic targeting — the sectionizer

Scoped editing presupposes a deterministic answer to “what is the character range of the target section.” Our sectionizer (~260 lines, zero LLM calls, zero dependencies) parses LaTeX source into a node tree along \section / \subsection / \subsubsection, assigning each node a char_start / char_end. Two design points deserve expansion:

Mask, don’t shift. A \section inside a comment or a verbatim environment is a false heading — but deleting those regions before parsing would invalidate every character offset. The solution is to replace them with equal-length whitespace:

def _mask(content: str) -> str:
 """Replace comments and verbatim environments with equal-length spaces
 (all character offsets preserved), so a \section inside them is not
 treated as a real heading — while splice still uses original offsets."""

Headings are located on the masked text; content is read from the original at the same offsets. Parsing and splicing share one coordinate system, so there is no translation step — and no translation bugs.

Dual anchors. Every node carries both a positional anchor (n0, n1, stable within one version) and a content-fingerprint key (L1:ErrorAnalysis, derived from level + title). Positional anchors drift when sections are inserted or removed; content keys follow the title and survive across versions, so they take precedence during resolution. This is the cheapest possible mechanism for “refer to the same section across versions”: no diff algorithm, just two string keys.

The failure mode is deterministic too: if no structure parses, the sectionizer returns editable=False and the caller falls back to whole-document revision. The boundary machinery never raises into the main flow — its job is to narrow the model’s scope, not to add a failure point.

Mechanism 2: scoped generation and deterministic splicing

After targeting, the prompt sent to the model contains only the target section (plus hard rules against scope escape), and the model returns only a replacement fragment for that section. The fragment’s path back into the master document is entirely code:

strip_fragment — cleanup: remove Markdown fences; if the model disobeyed and returned a full document, cut out the span from the first heading to just before \end{document};
check_fragment — health check: balanced braces, balanced environments, no \documentclass (soft signals that warn rather than block);
splice(content, start, end, new_text) — a one-line reassembly in which everything outside the range is byte-identical.

Byte-identical is a testable promise, not an adverb: the assertion is written directly into the test suite. One implementation lesson: a fragment must preserve the count of top-level \section commands within the replaced span, otherwise the positional anchors of subsequent edits in the same round drift and hit the wrong section. That bug was caught by an internal adversarial review and is now pinned by a permanent test.

Mechanism 3: zero-token pre-routing

“Which part does this feedback want to change” is a routing problem. The default is to ask a model, but a substantial share of feedback carries an unambiguous explicit target: “the bibliography format is wrong”, “the abstract is too long”, “the units in Table 2 are off”, “rewrite §3.1”. Those are decided by rules:

signal classes = { bibliography terms, abstract/keywords, section references
 (Table N / Figure N / § numbers / unique section titles) }
exactly one class hit → route directly (0 tokens)
no signal / conflicting classes / "whole document" wording → fall back to LLM routing

The design stance is conservative-first: prefer falling back to the LLM over ever guessing. Disambiguation is deterministic as well — when both “Data Processing” and “Data Processing and Results” match, the longer title wins. Combined with the narrow prompt from Mechanism 2, this cut input tokens for keyword-fix-type requests from roughly 6–7k to about 1.5k in our measurements; routing itself costs nothing, adds no latency, needs no API key, and runs at full speed in offline tests.

Mechanism 4: the zero-token regression gate

The autonomous agent’s reflection loop (draft → self-revise × N) is where scope escape does the most damage. Beyond constraining revision into a structured edit plan (schema-enforced changes[], where an empty array means converged and terminates early) plus scoped splicing, there is one final gate before anything lands — regression_check, six fully deterministic criteria:

loss of the compilable skeleton (\documentclass / \end{document});
length collapse below 60% of the previous version (truncation, or “rest unchanged”-style elision);
structural score decrease (eight equal-weight checks: section count, error analysis, data tables, equations, no TODO placeholders, …);
shrinkage of the \cite key set;
fewer sections;
fewer figures or tables.

Any trigger reverts to the previous version and stops the iteration. The reason for choosing deterministic criteria over one more LLM judge call is stated in the module’s own comments: this particular failure mode — losing content — is deterministically detectable, whereas the pairwise judge’s validity in this domain is unverified and would cost an extra call per round. More correct, and cheaper.

The gate has its own edge case: bibliography generation runs after reflection, so with an empty bib the “citations closed” check is false for any draft containing \cite, and the gate would misread “added a citation” as degradation. The fix excludes that check from the gate score while the bib is empty; citation loss is still caught separately by the key-set-shrinkage criterion. A gate is code, and code has bugs — but a gate’s bugs can be pinned by unit tests, while model drift cannot. That asymmetry is precisely why the gate belongs on the deterministic side.

Mechanism 5: putting the boundary itself under CI — prompt-shape gates

Offline tests cannot run a real LLM, so they cannot measure generation quality — but they can test the shape of a prompt: whether the scoped prompt still demands “change only this section”; whether the assembly function’s output on gap-free input is byte-identical to a golden file. If someone edits the prompt back to “output the full revised document”, plain pytest goes red. The most fragile part of the deterministic boundary — constraints written in natural language — thereby gets the same regression protection as code. A companion gate validates the scorer itself: gold-standard good reports must score ≥ 0.75, bad ones ≤ 0.35, with a gap ≥ 0.4 — the scorer must demonstrably encode quality before it is allowed to act as a gate.

The whole picture, and the measured gains

flowchart TD
 A[Edit request] --> B{Deterministic pre-router}
 B -->|explicit target hit| S[Scoped prompt: target section only]
 B -->|no signal / conflict| L[LLM router] --> S
 S --> G[LLM generates section fragment]
 G --> C[Deterministic cleanup and health check]
 C --> P[Splice: bytes outside range unchanged]
 P --> H{Who is in the loop}
 H -->|conversational editing| U[Proposal ledger: lands only on user accept]
 H -->|autonomous reflection| R{Zero-token regression gate}
 R -->|pass| V[Commit as new version]
 R -->|degraded| X[Revert to previous version, stop]

Mechanism	Alternative	Measured gain
Sectionizer + splice	Full-document rewrite by the model	Bytes outside the target range unchanged; avoids the 16–27% content-regression mode
Zero-token pre-router	LLM routing on every request	0-token decisions for explicit targets; input 6–7k → ~1.5k tokens
Regression gate	Accept every revision / LLM judge	One judge call saved per round; degradation reverts, convergence stops early
Prompt-shape gate	Manual review of prompt edits	The deterministic share of 351 tests runs fully offline, in CI

Where the principle does not apply

In keeping with this series’ convention, the honest boundary of the boundary:

It presupposes parseable structure. LaTeX has \section, code has ASTs, Markdown has headings; free-form prose has none. Without structural anchors there is no sectionizer — the boundary collapses to just cleanup and gates.
The boundary is code, and code has carrying costs. The sectionizer must handle starred headings, optional short-title arguments, comment and verbatim masking; we maintain dozens of unit tests for this layer alone. If the product’s shape is still changing fast, building the boundary early makes every change cost twice.
It protects stock, not flow. The boundary’s value grows with the value of existing assets: it prevents version N from being wrecked, and contributes nothing to generating version 1. Tasks that are creation-dominated, with no existing asset to protect, do not need it.
Deterministic criteria only measure the measurable. The regression gate prevents getting worse; it cannot certify getting better. Preference-level quality still requires pairwise judges or humans — which is the subject of another article in this series (validity engineering for LLM-as-judge).

References

Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (arXiv:2601.13217) — empirical measurement of the 16–27% content-regression mode in full-rewrite revision; structured edit plans and scoped revision are the remedies it names.
PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing (arXiv:2604.05018) — structured multi-agent pipelines beating autonomous baselines (+50–68% / +14–38%); its accept-or-revert refinement gate reports 79–81% win rate with 0% degradation (body-text detail, not independently verified).
Self-Refine: Iterative Refinement with Self-Feedback (arXiv:2303.17651) — feedback must be specific and actionable; generic feedback measurably hurts, and tasks where errors are hard to self-detect need external signals.
qwen-code issue #4695 — a public case of tool-loop collapse (43 identical calls / 8.9M tokens).

The testing pyramid for LLM applications: evals, not assertions

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 8 of the Engineering LLM Applications series. The evidence base is the same as the rest of the series: a production-grade AI writing agent (reads data → renders figures → generates LaTeX → compiles PDF), 351 test functions across the project.

The problem: non-deterministic output defeats assertions

The core move of conventional testing is asserting an exact value: assert f(x) == y. On an LLM main path this does not work — two generations from the same input are never verbatim-identical, so tests that assert fixed strings either stay red or get labeled flaky and lose all credibility. Teams commonly settle into one of two bad equilibria: test only the LLM-free edges and let the main path run naked, or force exact assertions and re-run until green. The real cost lands at change time: edit one word in a prompt and generation quality can silently degrade, with no signal in CI — part 11 lists this as failure mode four of a probabilistic component, “untestable regressions”.

The answer our project converged on fits in one sentence: an LLM application runs evals, not exact-value tests. Assertions are not abandoned — they change their object, from “the output equals X” to “the output satisfies property P”: structure score above a threshold, pairwise preference no worse than the previous version, a PDF that compiles, citations and numbers closed against their sources, metamorphic relations holding. Deterministic code keeps its exact assertions; the two kinds of test coexist in layers.

The philosophy: eval-driven development

Our methodology survey (docs/testing/01-research in the project, 2026-07, compiled from multi-source web search) compresses the field consensus into three points:

EDD (Eval-Driven Development): define the evaluation criteria before writing the agent; on every prompt/flow change → run the evals → read the score movement → decide. Evaluation is not post-hoc acceptance but the steering wheel of the development loop — TDD for agents.
The behavioral-testing triad (CheckList, ACL 2020): MFT (minimum functionality), INV (invariance), DIR (directional) tests. Transferred to this project: material with a planted gap must be flagged by the material check (MFT); deleting a data file must increase the gap count (DIR).
Metamorphic testing: when there is no reference answer, assert relations between inputs and outputs instead — “fuller material ⇒ structure score does not drop”, “delete half the data ⇒ more gap warnings”. This sidesteps the fundamental obstacle that non-deterministic output cannot be asserted exactly.

The tooling conclusion was no heavy dependencies: borrow DeepEval’s form — “an eval is a pytest case plus a threshold assertion” — and wrap our own scorers into pytest cases: zero new dependencies, offline-capable, self-hosted.

Four layers, split by “does it call a real LLM”

Layer	What runs	Calls LLM	Speed	How to run
L1 unit	Pure logic: parsing / indexing / routing / guards / sandbox whitelist / the scorers themselves / source verification	No	Seconds	`pytest`
L2 integration	Mock LLM drives the flow: full run_agent pipeline, tool loop, endpoint contracts	Mock	Seconds	`pytest`
smoke	Real local execution: matplotlib figure sandbox, xelatex compile	No	A few seconds	`pytest` (when xelatex is present)
L3 online eval	Real keys run the agent/judge; scorers plus thresholds decide	Yes (DeepSeek + Qwen)	Minutes	`pytest --run-live`

The layering criterion is not the textbook unit/integration split but two switches: does it call a real LLM, and does it need a real execution environment. A few design points:

How L2 is written: monkeypatch swaps agent._client for a fake, tool_calls and final content are scripted, and the test asserts flow events plus an eval_report score — it verifies “wired correctly”, not “high quality”. Note the scorer reuse: the same eval_report scores mock output in L2 and real output in L3, while being itself an L1 test subject — the scorer is tested before it is allowed to test anything else.
Tests never require compilation: pytest runs the Python source in seconds; the Nuitka build exists only to produce the customer-facing .exe. The dev loop = edit code → pytest → green.
Gating: @pytest.mark.live is skipped by default; only --run-live runs it, and missing keys auto-skip. The two real keys (DeepSeek for the text/agent main path, Aliyun Qwen for vision image reading) live in backend/.env, never in git. CI runs only the three offline layers by default; L3 runs manually, pre-release, or nightly.

pytest # L1+L2+smoke (default, seconds, offline, mandatory in CI)
pytest --run-live # add L3 online evals (real keys, slow, costs money)
python run_evals.py # one-shot golden-sample eval: scores vs threshold baseline

Objective criteria: five assertable properties

Agent output is never asserted against fixed strings; these properties are asserted instead:

Criterion	Implementation	Assertion form
Structure score	`eval_report.score_report`: 8 reference-free checks (compilable skeleton / ≥3 sections / error analysis / citation closure / data tables / equations / no AI boilerplate / no TODO), returns 0–1	score ≥ threshold
Pairwise preference	`eval_judge.judge_pairwise`: judged twice with positions swapped, ruled only if consistent	new version no worse than baseline
Compile pass	Real xelatex compile	PDF produced, page count > 0
Source closure	`verify.verify`: `\cite` closed against the bib, data numbers traceable to the material	no undefined citations; unsourced numbers bounded
Metamorphic relations	metamorphic / DIR	delete data → more gaps; fabricated number → more unsourced; fuller material → score does not drop

Two points deserve expansion. Pairwise preference must treat position bias: LLM judges systematically favor the first candidate (arXiv:2406.07791), so judge_pairwise evaluates twice with positions swapped and rules only on agreement; before judging anything, the judge must pass a reconstruction_accuracy self-check — correctly ranking known strong/weak pairs (≥ 0.7) — to earn the right to judge drafts. Validity engineering for the judge itself is the subject of part 9.

Metamorphic relations are the cheapest anti-hallucination criterion. The DIR test in the source-verification module, verbatim (string literals translated from the Chinese original):

def test_verify_metamorphic_fabricated_number_increases_unsourced():
 """DIR metamorphic: add a number absent from the material →
 the unsourced (suspected-fabricated) set must grow."""
 mat = "measured 1.0 and 2.0"
 base = verify.verify("values 1.0 and 2.0", mat)["numbers"]["unsourced"]
 more = verify.verify("values 1.0 and 2.0 and 7.77 from thin air", mat)["numbers"]["unsourced"]
 assert len(more) > len(base) and "7.77" in more

No knowledge of the “correct output” is needed — only a direction is asserted: one number appearing out of thin air must enlarge the unsourced set. The pattern replicates cheaply: delete a data file → more gap warnings; supply the .bib → the citation gap disappears.

The golden regression gate: the scorer must prove itself before it may gate

Online evals are the most faithful, but they need keys and take minutes — they cannot protect every change. So the regression gate is dual-track: beside the online gate run_evals.py (real generation plus judge), an offline gate test_golden_gate.py runs inside plain pytest, no keys needed, three gates:

Gate 1: scorer discriminative power. Gold-standard “good/bad” reports must be clearly separated by eval_report (assertion messages translated):

assert good["score"] >= 0.75, f"gold good report scored too low {good['score']}: {good['flags']}"
assert bad["score"] <= 0.35, f"gold bad report scored too high {bad['score']} (scorer lost discrimination)"
assert good["score"] - bad["score"] >= 0.4, "good/bad gap too small: scorer discrimination regressed"

The logical order matters: first prove the scorer actually encodes quality, only then allow it to act as a gate. A scorer whose discrimination has decayed renders every downstream threshold meaningless — and it will never raise an alarm about itself, which is why something else must pin it.

Gate 2: the gold-standard skeleton really compiles. The gold report skeleton must pass a real xelatex compile (skipped when no engine is installed locally; on the release machine / CI with TeX it is a hard gate). A generated report that fails to compile is the most direct signal of prompt or template regression; this gate protects the template plus the compile chain.

Gate 3: prompt shape. Offline tests cannot run a real LLM and cannot judge output quality — but they can judge the structure of a prompt: the scoped prompt build_user_scoped must still demand “rewrite only this section”, with \documentclass and \end{document} appearing inside the “strictly forbidden to output” clause; the full-document path build_user(gaps=None) must be byte-identical to the call without gaps. The moment someone edits the prompt back to “output the whole document”, plain pytest goes red.

(One additional foundation check: the gold-standard report must parse correctly under the deterministic sectionizer — protecting the base of section-level editing.)

The baseline: every score pinned to git_sha and prompt_hash

The online gate run_evals.py outputs more than red/green: every real run appends its scores to docs/testing/eval_baseline.tsv, and its exit code feeds CI. The file’s actual contents (fixture names translated):

date	git_sha	prompt_hash	metric	name	score	threshold	pass
2026-07-01 21:49	cb5bba9	6f3202abff3a	judge.recon	strong/weak pair	1.000	0.70	1
2026-07-03 11:29	39b9e29	46bed3f78ca1	report.score	field-sweep S-params	0.875	0.50	1
2026-07-03 11:29	39b9e29	46bed3f78ca1	report.compiles	field-sweep S-params	1.000	1.00	1
2026-07-03 11:29	39b9e29	46bed3f78ca1	judge.recon	strong/weak pair	1.000	0.70	1

Every row pins both git_sha and prompt_hash: when a score drifts, the first question — “did the code change or did the prompt change” — is answerable immediately. The full development loop:

flowchart TD
 A[Edit code / edit prompt] --> B{pytest: L1+L2+smoke, seconds}
 B -->|red| A
 B -->|green, quality-relevant code untouched| Z[Commit]
 B -->|green, agent/quality-relevant code touched| C[pytest --run-live or run_evals.py]
 C --> D[Real agent run → scorer/judge scores]
 D --> E{Compare against thresholds and eval_baseline.tsv}
 E -->|meets thresholds| F[Green: scores appended to baseline] --> Z
 E -->|regression| A
 Z --> R[Pre-release: default suite + --run-live all green → only then the Nuitka build]

Measured numbers from the rollout: on the day the system landed (2026-07-01), the default suite ran 144 passed + 7 skipped (live) and --run-live passed 7/7; two days later the golden gate was added, 248 passed with no regressions; at the time of writing the project holds 351 test functions. Flakiness governance comes down to two moves: LLM cases assert thresholds/intervals/metamorphic relations rather than exact strings, and the judge uses swap-averaging to reduce variance.

Where this does not apply

A small golden set is not a proof of quality. The survey notes that aggregate metrics need on the order of hundreds of examples to be trustworthy (that figure not independently verified); this project’s online golden set started with 1 report fixture plus 2 strong/weak pairs. At that scale the online eval is a smoke gate — it catches large regressions — not a quality metric. Do not conclude “quality improved” from it before the set grows.
Objective criteria only measure the measurable. The 8-item structure score is a reference-free floor: it judges “is this report solid and compilable”, not “is it well written”. Preference-level quality above the floor needs a judge — and the judge must be validity-tested first; an unvalidated judge gate is more dangerous than no gate.
Green L2 is not green quality. Mock-driven integration tests verify wiring and flow events; they know nothing about the quality of real model output. Reading L2 passes as a quality signal is the most common misreading of this layering.
Online evals are slow, cost money, and carry variance — keep them out of the fast loop. They belong pre-release, nightly, or after quality-relevant changes. And if the product is still in prototype phase with prompts rewritten daily, the threshold-maintenance cost of golden gates will exceed their value — establish a reasonably stable product definition first, then build the gates.

References

CheckList (ACL 2020) — the MFT/INV/DIR behavioral-testing triad; the direct source of this article’s criteria design.
arXiv:2406.07791 — systematic study of position bias in LLM judges (systematic preference for the first candidate).
arXiv:2410.15393 — judge calibration methods: swap-and-average, balanced-position calibration, and others.
arXiv:2504.18827 — metamorphic testing applied to LLMs (LLMorph and related work).
DeepEval — the “eval as a pytest case + threshold assertion” form we borrowed (without taking the dependency).

These external sources come from the project’s research document (2026-07, compiled from multi-source web search); apart from this project’s own measurements, second-hand claims were not independently re-verified.

The token economics of LLM applications

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 1 of the Engineering LLM Applications series. The series is grounded in the implementation and measurements of a production-grade AI writing agent (FastAPI + React + Tauri), and discusses transferable engineering methods.

The problem: why does fixing one keyword cost 6–7k tokens

When you build an LLM application, the first constraint that forces an architectural decision is usually not model capability — it is the bill. Our project (a LaTeX lab-report writing agent) received a blunt question from a user: “why did you send 6k?” — he only wanted to change the report’s keywords.

The precise ledger (before the fix):

Component	Content	Size (approx.)
System prompt	Role rules + report spec + writing methodology (1,948 chars) + golden-sample exemplar (1,235 chars) + LaTeX template rules	~3k tokens
Session material	First 6,000 chars of `material` — including the entire experiment txt bundle (data matrices, field-sweep parameters, …)	~3–4k tokens
Target block + instruction	One `\keyword{...}` line + the edit request	~0.1k
Total		~6–7k in / ~50 out

The root cause is not the model but one-size-fits-all context assembly: whatever the edit, the full set of quality anchors plus the full material gets stuffed in. Fixing a keyword has no use for a complete exemplar report about a different experiment, and even less for raw data matrices. The user’s follow-up became the design charter for this pipeline: “the process may be complex, but the goal is to save tokens.” Cost optimization is not cutting features; it is layering context by problem type — minimal context per type. Below are the four layers we shipped.

Layer 1: zero-token deterministic pre-routing

“Which part does this feedback want to change” is a routing problem, and the default is to ask a model. But a substantial share of feedback carries an unambiguous explicit target: bibliography, abstract, keywords, “Table 2”, “Figure 3”, “§3.1”, a uniquely matching section title. These are decided with regexes and string matching (_fast_route — zero LLM calls, no API key even needed): exactly one signal class hit → route directly; “whole document” wording, no signal, or conflicting classes → fall back conservatively to LLM routing, never guess.

The gain is more than one saved routing call: the decision has zero latency, is 100% explainable (the log is tagged “deterministic fast decision, 0 tokens”), and offline tests run at full speed. This pattern is developed further as part of the deterministic-boundary principle in Part 11.

Layer 2: tiered system prompts

The most expensive part of the system prompt is the quality anchors: the writing methodology plus the golden-sample exemplar, about 3.2k characters, roughly 2k tokens. They are indispensable for whole-document generation — an early refactor once dropped them and generation depth and style collapsed immediately — but useless for small edits like “fix one section” or “fix the keywords”.

So build_system(scoped=True) adds a tier: paragraph-level and metadata edits drop the quality anchors while keeping the role rules (no fabrication), the format spec, and the distilled judgment layer — the fragment must still be stylistically correct and compilable. Whole-document generation does not take this tier; its anchors stay. Each scoped call saves about 2k tokens. The full design of layered prompt assembly is the subject of Part 2.

Layer 3: material trimming and stable prefixes

The material layer does two things, and the second matters more than the first.

Deterministic trimming. select_material filters at zero token cost: pure-number matrix lines (bare CSV) are dropped; lines with lexical overlap with the target section or the feedback, or “data lines with textual annotation” (e.g. “measured 0.5 mm”), are kept, capped at 2,000 characters. The filter has a floor — it never filters the context away entirely:

# keep >=2 lines or >=40 chars; otherwise treat filtering as failed
# and fall back to the first `cap` chars
return out if (len(keep) >= 2 or len(out) >= 40) else m[:cap]

The whole-document revision path gets the same treatment (triggered only above 12k characters; never on first-version generation): autonomous exploration used to write up to 120k characters of raw files back into the material, after which every chat / whole-edit turn carried 40–70k tokens — even when the feedback was just “make the tone more formal”. After filtering: 8–15k per turn.

Stable prefixes. A deep audit produced an honest conclusion: on providers with prefix caching, a slimmer system prompt saves roughly nothing in direct cost (its value is focus — keeping irrelevant text from interfering); the real money is in the material block that misses the cache every turn. DeepSeek’s context cache is fully automatic, and the hit price is about 0.8% of a miss (v4-pro: hit $0.003625/M vs miss $0.435/M — vendor list prices verified online, 2026-07). To collect that discount, the material message must be byte-identical across turns — so the trimming and per-file digest functions depend only on the material itself, never on the current turn’s feedback, and the result is pinned as a fixed messages[0] prefix. Measured input cost: about -49%, multiplicative with the relevance filtering above.

The Anthropic path requires declaring the cache explicitly, and the TTL is a genuine trade-off:

system=[{"type": "text", "text": system,
 "cache_control": {"type": "ephemeral", "ttl": "1h"}}],

The reasoning is written in the source comment: users revise documents at human pace, so a 5-minute TTL expires between turns — you pay the cache-write premium every turn and never hit. The 1h tier writes at 2× price, reads at 0.1×, and every hit renews the TTL, matching the human rhythm. Below the model’s minimum prefix threshold it silently skips caching — harmless. Measured: Claude-path input cost down 54–65%.

Layer 4: per-step model tiering

Pipeline steps are not uniformly hard: the exploration tool loop (reading files, deciding what to read next), the material health check (JSON classification), BibTeX generation, and the English-abstract translation are simple, high-frequency tasks; drafting and reflective revision are what need the flagship model. So the model is passed per step: simple steps go to the cheap tier (DeepSeek V4 Flash — input $0.14/M, output $0.28/M, about 1/3 of Pro’s unit price; the vendor claims Flash matches Pro on simple agent tasks at roughly 12× lower cost — vendor claim, not independently verified), while drafting, reflection, figure generation, and compile self-repair stay on Pro. With no cheap tier configured, everything falls back to the main model — zero behavior change; and cost accounting accrues per actual model of each call, because a single flat rate misprices runs that mix the two tiers.

The measured ledger

Typical inputs after layering context by problem type:

Problem type	System prompt	Material	Typical in
Routing (explicit target)	—	—	0 (deterministic decision)
Routing (everything else)	Small router system	none	~1.5–2.5k
Edit abstract/keywords	Scoped tier	none (document outline digest instead)	~1.5k
Edit one body section	Scoped tier	relevance-filtered, ≤2,000 chars	~2–3k
Whole-document rewrite	Full (quality anchors intact)	full	10k+ (money well spent)

The opening example: ~6–7k input tokens before → 0 (fast route hits the keywords class) + ~1.5k after. The whole-document revision path drops from 40–70k to 8–15k per turn, then multiplies with the ~-49% prefix-cache discount. Compile self-repair was also switched to a fresh minimal context (current draft + error excerpt + template rules), saving 30–50k tokens per repair round.

Observation ships with the optimization: every call collects real token usage via usage_out (including cache-hit counts) into the process log and trace, so agent-lab can show “which step spent how much”. This round of changes added 10 backend tests (backend 290 passed, agent-lab 15 passed). One negative lesson: the accounting was once wrong — cost estimation did not discount cache hits, so autonomous-run costs displayed up to several times too high. Fix the observability first, or the optimization aims at the wrong target.

When not to do this

Never filter material for first-version generation. The material is the sole source of facts — err on the side of completeness; relevance filtering is for revision turns only.
Never slim the system prompt for whole-document generation. The quality anchors are where depth and style come from; that spend is worth it.
Never economize on the verification layer. Provenance checking still runs against the complete material server-side — you save on what is fed to the model, not on validation.
Nominal savings ≠ real savings. Once caching hits, many “big” optimizations shrink: splitting the exploration/figure contexts looked like 50k+ tokens saved, but the cache already absorbed about 90% of it, leaving 10–20% real savings — we downgraded that item accordingly. Rank optimizations by post-cache-discount numbers.
Fix observation before optimizing. When accounting ignores cache hits, the most expensive path looks even more expensive than it is, and steers the optimization effort in the wrong direction.

Validity engineering for LLM-as-judge

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 9 of the Engineering LLM Applications series. The first-hand data comes from a writing-judgment distillation experiment (whose design and headline results are the subject of Part 12); every number is taken from that project’s experiment journal, run ledger, and source code.

The problem: the judge is itself an unmeasured probabilistic component

Pairwise judging (pairwise LLM-as-judge) is the mainstream way to evaluate LLM application quality: show the model drafts A and B, ask which is better, tally win rates. It sidesteps the calibration problems of absolute scoring — and introduces a subtler one: the judge is a probabilistic component too. Why should its verdicts be trusted?

This is not a hypothetical risk. It happened twice in our experiment. The primary judge of the first round was llama-3.3-70b; measured after the fact, its QWK against the quality anchor was 0.10 — near random — and the entire round was voided. In a later round, claude-sonnet-4.6 ruled B>A while gpt-4o ruled A>B on the same drafts; the aggregate verdict could only be recorded as “ambiguous”. If you cannot answer “which judge is credible”, the win-rate table is just another array of random numbers.

This article names the practice we converged on validity engineering: before consuming any verdict from a judge, establish with independent evidence that the judge measures what you think it measures. It consists of five mechanisms: admission, anchor independence, scale diagnosis, bias control, and a pre-deployment acceptance gate.

Mechanism 1: judge admission — human-score anchors and a QWK bar

Admission works like this: have each candidate judge score a set of anchor texts that carry quality scores, then compute QWK (Quadratic Weighted Kappa, an ordinal-agreement metric) against the anchor scores. The bar is written directly into the module docstring of qwk_anchor.py: “≥0.5 → the judge is credible and pairwise win-rate conclusions are worth believing; <0.3 → go back and fix the rubric first”, with 0.3–0.5 recorded as “marginal”.

Measured results for the four candidate judges:

Judge	QWK	Ruling
claude-sonnet-4.6	0.55–0.61	Retained (primary judge)
qwen3.7-max	0.571	Retained (cross judge)
gpt-4o	0.281	Eliminated (<0.3, not credible)
llama-3.3-70b	0.10	Eliminated; the first round it served as primary judge was voided in full

Elimination is not academic fastidiousness: gpt-4o was one party in the “two judges, opposite verdicts” incident above, and removing it returned the ruling to the discriminative power of the credible judges. But this round of QWK was computed against a synthetic anchor — which leads to the next mechanism, and the most expensive lesson in the whole system.

Mechanism 2: the anchor must be independent of the system under evaluation — a circular-validation incident

The incident was triggered by one user question: “is the test data even representative?” The audit found that none of the three real datasets the plan specified (arXivEdits / peS2o / ASAP) had ever been downloaded successfully — a hardcoded URL had long 404’d, the Kaggle login step was never done, and the download script ignored the system proxy. Everything had run on seed and synthetic fallback data: 13 exemplars, 5 hand-written gold pairs, 8 synthetic pairs, and a 39-item anchor whose scores were synthetic, not human.

The consequence was structural: the so-called “QWK anchor” was in fact the judge aligning with the program’s own degradation rules — circular self-validation. Historical QWK of 0.51–0.61 was voided in full; it constitutes no human-grounded credibility whatsoever. One asymmetry is worth stating: what the incident voided were the “pass” rulings; the “out” rulings actually stand — a judge that cannot even align with the coarse ordering of a synthetic anchor is even less likely to align with humans (our inference, not an independent experiment).

The fix was to actually connect the real data: 200 genuine expert-revision pairs from arXivEdits (filtered by the intention field for writing-quality edits, discarding trivial typo fixes; median length difference 27 characters), and 120 essays with real human holistic scores from ASAP 2.0 (stratified sampling, 20 per score band 1–6, guaranteeing QWK has range to work with). The lesson in one sentence: the anchor must come from outside the system under evaluation, or QWK measures the judge’s agreement with your own code.

Mechanism 3: joint QWK×Spearman diagnosis — scale compression is not ranking failure

With the human anchor in place, the first real QWK came out at 0.323: model mean 2.32 vs human mean 3.50, a systematic severity bias of about 1.2 points. A multi-agent investigation diagnosed three root causes: severity-biased scale misalignment; a genre/rubric mismatch (a physics-report rubric grading English argumentative essays); and the inherently low ceiling of single-essay absolute scoring as a usage mode (the project’s investigation cited roughly 0.60 as the zero-shot ceiling for ASAP holistic scoring — a reference value, not independently verified).

The fix took the zero/low-cost route: switch to the ASAP holistic six-band rubric, delete all “strict” wording (the system prompt changed from “strict” to “well-calibrated”), add a full-range prior; few-shot examples stayed disabled behind a comment, because the anchor data had no train/eval split and would leak. Before and after:

Metric	Before	After
QWK	0.323	0.456
Spearman	—	0.813
Pearson	—	0.788
Systematic bias	-1.18	-0.87
Model mean	2.32	2.63 (human 3.50)

QWK still did not clear 0.5 — but the key finding sits in another number: Spearman 0.813 ≫ QWK 0.456. The diagnostic rule in qwk_anchor.py summarizes it in one line: “high Spearman with low QWK → a pure scale problem (recalibration can save it); both low → the ranking signal itself is weak.” The confusion matrix supplied the hard evidence — the judge never awards a 5 or a 6; all 40 essays humans scored 5 or 6 were compressed into the 3–4 columns. This is a strong-ranking, conservative-scoring judge, not one scoring at random.

The conclusion lands on usage: pairwise judging consumes only the ranking (the judge’s strength), never the absolute scale (its weakness) — so pairwise win rates stand on solid ground as the primary signal, while single-essay absolute scores are unusable. The scale can be partially corrected by post-hoc monotonic recalibration, but its legitimacy has three preconditions (confirmed by adversarial verification): the mapping must be monotone, numbers must be reported on held-out/CV data, and parameters must not be transferred across domains — a 5-fold reproduction showed in-sample recalibration inflates the number and held-out evaluation automatically strips the inflation away. And recalibration cannot rescue the top of the scale: humans’ 5s and 6s are completely indistinguishable to the judge (both get 3s and 4s) — a genuine ceiling.

Mechanism 4: bias control — position swapping, identity masking, a rubric that penalizes verbosity

LLM judges have position preferences (favoring whichever draft appears first or last). The countermeasure is protocol-level: every pair is judged twice, in the orders (A,B) and (B,A), and if the two rulings disagree, the pair is recorded as a tie — never a fake verdict. The core logic is four lines (eval_judge.py):

sys_ab = _MAP_AB[_one_order(domain, a, b, rubric, cf)] # order 1: first=A
sys_ba = _MAP_BA[_one_order(domain, b, a, rubric, cf)] # order 2: first=B
consistent = (sys_ab == sys_ba)
decided = sys_ab if (consistent and sys_ab != "tie") else "tie"

The design yields a judge-quality signal for free: swap consistency. Measured, claude-sonnet-4.6 was swap-consistent on 93% of pairs at N=30 (28/30), and 8/8 in the N=8 pilot; qwen3.7-max mostly ranged 0.2–0.5, its win rates dominated by ties — the protocol automatically absorbs a weak judge’s instability into ties instead of letting it contaminate conclusions.

Two companion measures: identity masking (the judge cannot see which pipeline produced which draft), and length bias treated by rubric rather than by hope — the judging prompt states explicitly that “longer is not a virtue; low information density is penalized as verbosity; when substantively equal, the longer one must not win for being longer.”

Mechanism 5: reconstruction acceptance — judge known strong/weak pairs before judging new drafts

Beyond admission and diagnosis there is one final gate before deployment, called “ICAI-style acceptance” (reconstruction) inside the project: hand the judge known strong/weak pairs (gold-standard good drafts vs deliberately degraded weak ones) along with its rubric and have it pick “the better one”; only a pick-correct rate ≥ 0.7 (the code constant GATE_THRESHOLD = 0.7) qualifies it to judge new drafts, with swap consistency reported alongside. The logic is plain: a judge that cannot pick correctly when the answer is known has no standing to rule when it is not.

The cost boundary is also written into the module docstring: two swap calls per pair are real LLM calls and burn tokens — so this gate lives on the development/offline side and does not run in the client’s real-time path by default.

The whole picture: a judge’s path to deployment

flowchart TD
 A[Candidate judge] --> B[Score human-anchor set → QWK]
 B -->|QWK < 0.3| X[Eliminated]
 B -->|clears the bar| C{Joint QWK × Spearman diagnosis}
 C -->|ranking weak too| Y[Fix rubric / few-shot, re-test]
 C -->|ranking strong, scale compressed| D[License pairwise judging only<br/>absolute scoring unusable]
 D --> E[Reconstruction gate: known-pair pick-correct rate ≥ 0.7]
 E -->|fail| Y
 E -->|pass| F[Deploy: swap-order double judging<br/>disagreement = tie, identity masked, rubric penalizes verbosity]

Where this does not apply

In keeping with this series’ convention — the limits of the practice, and what it does not solve:

Same-family judge self-preference must be declared. Our primary judge, claude-sonnet-4.6, belongs to the same family as the Claude Opus models that performed skill distillation and adversarial auditing; every “strong” conclusion of the experiment comes from this same-family judge, carrying a “Claude recognizes Claude-style reasoning” self-preference risk. There are only two ways out: a third-party judge from a different family, or human-anchor re-verification.
The anchor’s domain is not the target domain. An English anchor only certifies the judge on English argumentative essays; it certifies nothing about judging Chinese physics reports — an external-validity gap. The production-side port hard-codes this warning in its docstring: the pairwise judge’s reliability in the Chinese domain is unverified, and it does not run in the real-time path by default.
High-band discrimination is unverified. Spearman 0.81 is driven mainly by the mid-to-low score range; yet pairwise comparison happens precisely in the “neither draft is bad” region — the judge’s ranking ability there is an untested assumption, and the upper-range collapse is a real threat to it.
An LLM judge does not replace a human anchor. When all preference evidence comes from LLM judges, systematic LLM-judge preferences cannot be ruled out; the experiment’s human blind evaluation (30 pairs) was still being collected at the time of writing, and is honestly listed as the most critical limitation.
Do not use a judge for deterministically detectable failures. Content loss and structural degradation are handled by a zero-token regression gate (see Part 11) — more correct, and one judge call cheaper per round. The judge belongs only at the preference layer, where deterministic criteria cannot reach.

References

arXivEdits — a dataset of genuine expert revisions (schema: sentence-1/sentence-2 + intention); the project filtered by intention for writing-quality edits, taking 200 pairs as strong/weak pairs.
ASAP-AES 2.0 — the English essay benchmark with real human holistic scores (1–6); obtained via the Hugging Face mirror jatinmehra/Automated-Essay-Scoring-2.0, with 120 essays stratified-sampled as the credibility anchor.
First-hand records: every number in this article comes from the project’s experiment journal and machine-readable run ledger (journal / runs.tsv); per-essay (id, human, model) scores are persisted to results/qwk_scores.csv to support recomputation and recalibration.

Wiring external signals into the loop: compile-and-repair

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 6 of the Engineering LLM Applications series.

The problem: models cannot fix errors they cannot see

Scenario: an autonomous agent generates a LaTeX lab report, and xelatex fails. Who fixes it?

The reflex is to ask the model to “check it again.” The evidence against that path is specific. One of Self-Refine’s conclusions (arXiv:2303.17651) is that on tasks where errors are hard to self-detect, pure self-reflection is nearly useless, and gains only return once an external verification signal is wired in. The systematic review in When Can LLMs Actually Correct Their Own Mistakes (TACL 2024) is blunter: prompting-only self-critique does not improve — and can degrade — arithmetic and code tasks, and the flattering self-correction numbers in the literature mostly leaned on an oracle (stopping only when the known-correct answer appears). Compile errors are the canonical hard-to-self-detect case: the model rereads the \def line it just wrote and cannot see that a % inside the braces comments out the closing brace — but xelatex sees it, and its error report cannot be faked.

So the right question is not “how do we make the model self-check” but “how do we wire an objective external verifier into the generation loop.” This article dissects the compile-and-repair loop we shipped in an AI writing agent — really compile after generation; on failure, extract the errors and feed them back for a repair round — and the engineering constraints that keep this deceptively simple loop under control in production.

From patching to really compiling

Before the compiler was wired in, the system already had a deterministic sanitizer layer (part 3 of this series): \mathbf→\symbf, % comments inside \def lines, backfilling template variables the model forgot to set — the known pitfalls that rules can encode. But patches cannot enumerate the long tail. The devlog states the goal of this step plainly: “not patching with the sanitizer, but having the agent really compile after generating the report, feed the xelatex errors back for a round of its own fixes on failure, and push the first-pass success rate to the max.”

That fixed the division of labor: rules handle the known pitfalls (zero tokens); the compiler backstops the long tail (one LLM call per round). In practice the sanitizer has already cured the common traps, so the first compile check usually passes outright; the repair branch only catches what remains. The loop itself is three steps:

compile_tex — assemble a complete working directory (article.tex + cls + bib + figures), run xelatex → bibtex → xelatex ×2, return {ok, pdf_path, log};
extract_errors — pull from the log the lines starting with !, the l. line-number lines, and Runaway / Emergency stop lines, capped at 1800 characters — never feed back the whole log;
on failure, feed the error excerpt back to the model, demand the full revised source, at most 2 rounds.

The cap of 2 is not arbitrary: a survey finding (How Many Tries, arXiv:2604.10508) across 7 models is that the first two rounds capture 76–95% of the achievable gain, with near-zero return from round three on; Self-Refine likewise plateaus by round 3. Extra rounds only multiply wall-clock and token spend. One further detail: the second round’s repair output is not re-verified inside the loop — the repaired draft is kept, and final verification is left to the user-side manual compile/export path.

A counter-intuitive design point: the repair is a full-document rewrite, not a scoped patch of the kind used in part 7. The reason is a property of LaTeX errors — they are often decoupled from the root cause, and line numbers are unreliable (LaTeX Compilation Challenges, arXiv:2603.02873), so pinpoint fixes at the reported line easily patch the wrong place. Scoped editing presupposes deterministic targeting, and compile errors are precisely where that presupposition fails.

Termination: fingerprint, round cap, wall clock

The biggest production risk of a self-repair loop is not failing to fix — it is failing to stop. The consensus on loop governance is that “when to stop” must not be left to the model’s discretion; an outer controller enforces it with deterministic criteria. There are three gates:

The no-progress gate. After each failed round, fingerprint the errors — the first 120 characters of the first ! error line (LaTeX’s anchor line); if it equals the previous round’s fingerprint, declare no progress, stop immediately, keep the current draft:

def _error_fingerprint(errs: str) -> str:
 """Compile-error fingerprint: first '! …' error line (LaTeX's anchor),
 else the first 120 chars. Used for no-progress detection."""
 for line in (errs or "").splitlines():
 s = line.strip()
 if s.startswith("!"):
 return s[:120]
 return (errs or "").strip()[:120]

If the same error survived a repair round, this model has no answer for this error; feeding it the same report again will most likely produce the same failure, and burning more tokens has no expected return.

The round cap. for attempt in range(2) — a hard limit.

The wall-clock gate. A single compile runs up to 4 subprocess passes at 180 seconds each — a token budget cannot contain CPU time. COMPILE_WALL_BUDGET (default 600 seconds) is checked before each round; on expiry, stop and keep the current draft. The config comment states the motive: prevent multi-round self-repair from hogging the customer machine’s CPU.

All three gates are deterministic criteria executed on the Python side; the model has no say in whether the loop stops — the same stance as the tool-loop circuit breaker in part 4.

Minimal repair context: ~8k tokens, not 40–60k

How the repair call’s context is constructed is the single most money-saving decision in this loop. The intuitive approach is to reuse the agent’s full exploration history — surely the model “fixes better knowing more”? The opposite holds. The judgment recorded in the code comment: 95% of compile errors are template-level syntax problems (balancing, command misuse, template variables), and fixing them requires exactly three things:

the full current draft;
the error excerpt (≤1800 characters);
the formatting hard rules (no % comments inside \def braces; bold vectors with \symbf; no bold inside accents; balance braces/environments/$; end with a top-level \end{document}).

About 8k tokens in total. Reusing the full exploration messages would carry 40–60k tokens of dead weight — source material, tool-call records, multiple draft versions — none of it contributing to fixing one brace; switching to a fresh minimal context saves 30–50k tokens per repair round. The system prompt narrows the mandate to a single sentence:

You are a LaTeX compile-repair assistant: fix compile errors only — do not rewrite content, do not add or remove sections, do not touch data.

This is not only about cost: the measurements in Revisit Self-Debugging (arXiv:2501.12793) show label-only feedback beating verbose detailed feedback — inaccurate long context introduces ambiguity and lowers scores. Commercial products are isomorphic: Overleaf’s commercial Error Assist also deliberately minimizes the repair context, sending only the full error, the relevant code lines, and a file-name list (product-page information, not independently verified).

Concurrency isolation: a compile directory per session

A plain constraint you only learn by tripping on it: the compile working directory must be isolated per session. compile_tex writes article.tex into the directory, copies the cls and figures, and rmtree-rebuilds the figure subdirectory during assembly — two concurrent generation runs sharing one directory will overwrite each other’s source, delete each other’s figures, and pollute each other’s logs. After partitioning the repair directory as _repair/<session_id>, each run writes its own. The lesson generalizes: a verifier is a side-effecting process, not a pure function — when wiring an external verifier into a loop, isolation is a default, not an optimization.

The whole picture, quantified

flowchart TD
 A[Final draft] --> W{Wall-clock budget exceeded?}
 W -->|yes| S[Stop, keep current draft]
 W -->|no| C[compile_tex: real compile]
 C -->|ok| P[PDF produced]
 C -->|fail| E[extract_errors: pull ! error lines]
 E --> F{Fingerprint same as last round?}
 F -->|same = no progress| S
 F -->|different| R[Minimal-context repair call ≈8k tokens]
 R --> N[2 rounds used?]
 N -->|yes| S
 N -->|no| W

Constraint	Failure mode without it	Implementation and numbers
Round cap	Grinding on for rounds with near-zero return from round 3	K=2; first two rounds capture 76–95% of achievable gain (survey figure)
No-progress gate	The same error fed back again and again, spinning in place	Fingerprint = first 120 chars of the first `!` line; stop on consecutive match
Wall-clock gate	4 passes × 180s per compile; a token cap cannot contain CPU	`COMPILE_WALL_BUDGET`, default 600s; on expiry keep the current draft
Minimal context	40–60k tokens of exploration-history dead weight per call	Current draft + error excerpt + hard rules ≈8k; saves 30–50k tokens per round
Session isolation	Concurrent runs clobber each other’s source/figures/logs	Dedicated `_repair/<session_id>` working directory

Where not to build this loop

It presupposes a cheap, objective, unfakeable verifier. Compilers, type checkers, and unit tests qualify; “is the writing good” does not. The stop condition must anchor on an objective signal like “compilation succeeded” — never on the model’s self-assessment. Quality dimensions without a verifier belong to evaluation and judges (parts 8 and 9), not to this loop.
Deterministic repair comes first. Pitfalls that can be written as rules (known syntax traps, missing-figure placeholders, variable backfill) should be cured by the zero-token sanitizer before any LLM repair; the repair loop only backstops the long tail that rules cannot enumerate. Reversing the order means using a probabilistic component for deterministic work: paying a call every time while adding new uncertainty.
Weak models need a non-LLM exit. Survey data puts weak models’ repair rate on deadlocked errors at only about 30% (not independently verified). The loop must ship with a fallback that does not depend on the model — here, export_zip produces a compilable package to hand to Overleaf or a human. Without an exit, the loop degenerates on weak models into “burn the budget, then give up.”
When the verifier is expensive, the budget gate precedes the round cap. Here a single compile is wall-clock-bounded at 4 passes × 180 seconds (at most 720s), so K=2 is affordable; if the verifier is a ten-minute integration test suite, the same K=2 may be unacceptable — the wall-clock budget must become the first gate, including the option of “give up without repairing at all.”

References

Self-Refine: Iterative Refinement with Self-Feedback (arXiv:2303.17651) — self-reflection is nearly useless on tasks where errors are hard to self-detect; gains return with external signals; benefit concentrates in the first 1–2 rounds (its “+20% average” figure not independently verified).
When Can LLMs Actually Correct Their Own Mistakes (TACL 2024) — prompting-only self-critique does not improve, and can degrade, arithmetic/code tasks; flattering self-correction results mostly relied on an oracle.
How Many Tries (arXiv:2604.10508) — the first two rounds capture 76–95% of achievable gain; near zero from round 3.
LaTeX Compilation Challenges (arXiv:2603.02873) — LaTeX errors are often decoupled from root causes; line numbers are unreliable.
Revisit Self-Debugging (arXiv:2501.12793) — label-only feedback beats verbose feedback.
Overleaf Error Assist — the commercial isomorph of minimal repair context (product-page information, not independently verified).