LLM | kaguc — Writing to understand systems.

Distilling judgment, not structure: a controlled experiment

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 12 of the Engineering LLM Applications series.

The hypothesis: can distilled judgment beat a distilled template?

Hand expert exemplars to a model and ask it to induce a reusable writing skill (a prompt), and there are two routes:

Path A (structural template): extract “how many parts, what goes into each, in what order, in what format”;
Path B (judgment logic): contrastive rationale extraction — show the model strong and weak versions of the same task and force it to articulate why the strong one is better, what it assumes about the reader, how a novice gets it wrong; aggregate the output into judgment rules + do/don’ts + a reader model.

H1: the Path-B skill beats the Path-A skill in pairwise preference judging. Everything is inference-only — no training, no fine-tuning, LLM APIs all the way down. Most prompt-engineering practice defaults to Path A; if H1 holds, there are many places where the recipe should change. This article is a full experiment report, including three methodological corrections that each came close to voiding the conclusion.

Design: control everything except one variable

All data comes from public benchmarks. Exemplars and strong/weak pairs are drawn from ASAP-AES 2.0 (English argumentative essays, human holistic scores 1–6): exemplars are score-6 essays; strong/weak pairs contrast score 5–6 against score 1–2 whole essays. A further 120 human-scored essays (20 per band) serve as a judge-credibility anchor, and the ablation uses arXivEdits same-sentence before/after expert rewrites.

The control discipline: A and B share the same exemplar input, the same generation model (DeepSeek-v4-pro, T=0.7), the same tasks, the same length constraint (roughly 250–400 words, no padding), and the same judges — the only variable is the distilled skill. Three bias controls on the judging side: position swapping (a win counts only if both presentation orders agree, otherwise it is a tie), identity masking (judges see only “Essay One / Essay Two”), and length neutrality (the rubric explicitly ignores length and penalizes verbosity). Dual judges: Claude-Sonnet-4.6 (primary) + Qwen3.7-max (cross-check).

The two distilled prompts are distinguishable at a glance. skill_A is position-and-proportion instructions throughout; a typical entry:

Introduction (1 paragraph) … A clear thesis statement that takes a definite position. … Relative length: 3–5 sentences.

Every entry in skill_B carries a “for whom, and why”:

When you acknowledge a counterargument, respond with a genuine rebuttal – not dismissal – because your reader expects fairness and intellectual seriousness.

Statistical lesson 1: effective N ≈ 1

The first round distilled a single pair (skill_A, skill_B), generated on 30 tasks, judged pairwise, and ran a binomial test treating the 30 pairings as independent samples: p ≈ 7×10⁻⁹. A beautiful number built on a broken inference: those 30 pairings share one and the same skill pair — only the tasks vary. What the test measures is “how stable this one skill pair is across 30 tasks”. H1, however, is a claim about methods, and the independent unit for that claim is the skill — of which there was exactly one pair. Effective N ≈ 1.

The redesign: first expand the exemplar pool from 12 to ≥24 essays, then use distinct random seeds to draw 6-exemplar subsets (Path B draws matching strong/weak-pair subsets), distilling K=10 independent skill pairs. Each pair generates and is judged on a shared task set (15 pairings per pair entered the statistics as executed), yielding one preference score w_k per pair; the test then runs at the skill level — Wilcoxon signed-rank on whether the K values of w_k are systematically above 0.5, with confidence intervals from a cluster bootstrap that resamples by skill. At the redesign’s budgeted scale this comes to 1200+ LLM calls across generation, judging, and distillation — roughly 5–6× the first round. Statistical honesty costs money.

flowchart LR
 P[Exemplar pool ≥24] -->|seed k draws 6 exemplars + pair subset| D[Distill K=10 pairs<br/>skill_A_k and skill_B_k]
 D --> G[Shared tasks ×2 generations per pair<br/>same model · T=0.7 · 250–400 words]
 G --> J[Dual-judge pairwise eval<br/>position swap · masking · length-neutral]
 J --> W[One preference score w_k<br/>per pair per judge]
 W --> S[Skill-level Wilcoxon<br/>+ cluster bootstrap CI]

Statistical lesson 2: metrics degenerate silently

A measurement check before the redesign saved the entire test: across 55 decided pairings under two judges and two rounds, A had won zero times — all the variation lay between “B wins” and “tie”. Define w_k = B/(B+A) as intuition suggests, and with A ≈ 0 almost every w_k collapses to exactly 1.0 — K samples with zero variance, and the Wilcoxon test degenerates into measuring nothing. The fix is a tie-inclusive preference score:

w_k = (B wins + 0.5 × ties) / T

Ties pull w_k toward 0.5 and restore real variance: on the original single pair this measures ≈0.97 under Claude and ≈0.73 under Qwen — the effect-size gap between the two judges is preserved and becomes testable.

Results: direction rock-stable, effect size judge-dependent

Judge	w_k mean	median	std	95% CI (cluster bootstrap)	w_k>0.5	Wilcoxon two-tailed p	old “per-pairing” p (overestimate, do not cite)
Claude (primary)	0.830	0.833	0.143	[0.747, 0.913]	10/10	0.0020	7.95e-24
Qwen (cross)	0.617	0.617	0.086	[0.567, 0.667]	9/10	0.0039	2.06e-06

Four notes that must be read alongside the table:

The old per-pairing p values are flagged in the statistics script’s own output: “overestimate: K×T pairings are non-independent; shown for contrast only, do not cite”. Landing a known-misleading number on disk together with its warning is more reliable than clarifying it verbally afterwards.
p=0.0020 is a floor, not an effect size. When all K=10 pairs point the same way, the exact two-tailed Wilcoxon p bottoms out at 2/2¹⁰ ≈ 0.0020 — equivalent to a sign test. It encodes only “10 pairs, one direction”, not by how much: w_k values of all-0.51 and all-0.99 yield the identical p, and p<0.001 is unreachable before K≥17. Effect-size evidence lives in the CIs.
Qwen’s “significance” carries less weight: tie-dominated, swap-consistency mostly 0.2–0.5, CI lower bound 0.567 hugging 0.5. But a decided-only sensitivity analysis (ties removed) still shows 9/10 above 0.5 with zero net losses — the significance is not manufactured by tie-weighting.
A is not winless: A won individual pairings in 4 pairs under Claude and 8 under Qwen — but B suffered a net loss in no pair under either judge.

Ablation: does the judgment signal survive same-sentence pairs?

B’s contrast source is cross-topic whole-essay pairs — its wins could be contaminated by topic/content differences rather than pure “judgment”. Swapping the contrast source for arXivEdits same-sentence rewrite pairs (near-zero topic difference) yields B_sent, run over the same 10-pair protocol:

Judge	B_sent w_k mean	95% CI	pairs >0.5	vs A (Wilcoxon p)	B vs B_sent
Claude	0.720	[0.663, 0.773]	10/10	0.0020 (significant)	+0.110, p=0.186 (n.s.)
Qwen	0.567	[0.493, 0.640]	6/10	0.109 (not significant)	+0.050, p=0.281 (n.s.)

The reading stays conservative: under Claude, B_sent still significantly beats A and is statistically indistinguishable from B, suggesting the advantage stems from judgment rather than topic. But Qwen does not reproduce the ablation, and a genre confound remains — arXivEdits are scientific-paper sentences while the generation task is argumentative essays, so “same-sentence” and “genre” changed simultaneously; the clean single-factor control has not been run.

Judgment purity: an adversarial audit revises the self-assessment down

One question could still overturn the whole narrative: are skill_B’s entries genuine judgment rules, or structural descriptions in new clothing? LLM self-assessment cannot be trusted — the pilot’s sharpness self-check reported 100%. Instead, an 11-agent adversarial audit (executed by Claude Opus 4.8: classify every entry as judgment/structure, cross-compare against the paired skill_A for re-skinning, and specifically attack “is the because a tautology?”) spot-checked 5 pairs:

Pair	Initial self-ratio	After adversarial review	Overturned entries (re-skinning suspects)
1	87%	80%	context-in-opening ≈ A’s Intro recipe
2	82%	73%	“vivid example” ≈ show-don’t-tell tautology
3	73%	67%	ordering by persuasiveness ≈ structural organization
4	82%	64%	rebuttal / closing ≈ A’s counterargument / conclusion sections
5	100%	90%	point ordering = an organization rule

After adversarial review the mean drops to 0.746, yet all 5/5 pairs remain judgment-dominated and substantively distinct from skill_A (which is position/format instructions throughout, with zero reasoning). Verdict: “pass (with conditions)” — 0.75 is an upper bound that includes redundant clusters, and pair 4 is the weakest link (0.64). The overturned entries share one pattern: a structural action glued to a near-vacuous because. Note that this remains automated AI review, not independent human verification. (Adversarial review as an engineering practice is the subject of Part 13, Multi-agent adversarial review as an engineering practice.)

Seven honest limitations

No human anchor collected (the most fatal). A 30-pair blind-evaluation list has been exported (the machine ruling on the same pairings: B=20 / A=3 / tie=7), but human scores are not yet in. Until then, every piece of preference evidence comes from LLM judges, and a systematic LLM-judge preference cannot be ruled out.
Judge–distiller family overlap. The primary judge, Claude-Sonnet, shares a model family with the Claude Opus that performed skill distillation and the purity audit; every “strong” result happens to come from that same-family judge — a self-preference risk.
p bottomed out; effective N still optimistic. p=0.0020 encodes only “10/10 same direction”; and the 10 subsets, drawn from a finite pool, necessarily overlap — “effective N=10” is not strictly i.i.d. and remains an overestimate.
Effect size is judge-sensitive. Claude 0.83, Qwen 0.62: “B does not lose to A” is solid; “how much better B is” is not settled.
Judge range collapse. QWK=0.456 (Spearman 0.813), and the confusion matrix shows the judge never awards a 5 or 6 — discrimination at the high end is unverified, precisely the region where B vs A (“both essays are decent”) is being compared. The systematic treatment of judge validity is Part 9, Validity engineering for LLM-as-judge.
Single-judge ablation + genre confound. The same-sentence ablation is significant only under Claude, and it mixes in the scientific-sentence → argumentative-essay genre variable.
Mechanism and external validity. The secondary signal — rhetorical move coverage, A 0.89 vs B 0.86 (a weak negative conclusion: no evidence that B wins by stacking moves) — comes from the pilot, was not re-measured in the main experiment, and carries no statistical test; and only one generation model and one genre were tested.

The conclusion therefore freezes as: under the LLM-as-judge lens, “learn judgment > learn structure” receives preliminary support that is method-level, directionally highly consistent, and purity-checked (with conditions); H1 is a well-evidenced candidate, not an established result. Establishing it requires human evaluation, a third-party judge, and a clean same-sentence same-genre ablation.

When not to do this

If a deterministic metric exists, skip the pairwise judges. This heavyweight statistical apparatus is only warranted where quality can only be measured by preference comparison; wherever an assertion can be written, one line of assertion beats 800 judging calls.
Path B presupposes strong/weak pairs. Contrastive distillation extracts rationale by confronting strong and weak versions; in domains with no natural quality banding (such as human scores) and no credible synthetic degradation, Path B has nothing to contrast.
If the budget cannot afford K pairs, do not claim a method-level conclusion. Single-pair results support only “B beat A on this pair”; treating the task count as the sample size was this experiment’s earliest mistake — and the easiest one to make.
Structural templates are not useless. Path A matches B on rhetorical move coverage (0.89 vs 0.86); when the task’s bottleneck is format compliance rather than quality judgment, Path A is cheaper and steadier.

Document versioning and scoped editing

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 7 of the Engineering LLM Applications series. The evidence base is the same as in earlier parts: the implementation, devlogs, and test suite of a production-grade AI writing agent (FastAPI + React + Tauri).

The problem: the model edits the document — what guards the document?

A typical incident from live testing: the user types feedback like “polish this”, “fix the bibliography”, “change Table 1” into the chat box, and some of these silently rewrite the entire document — the result lands directly as a new version, and the user discovers only afterwards that the change went far beyond what they asked for. In this class of application, the document is the user’s core asset, and every model output can touch it. Translating the problem from “model behavior” into “data model”, the requirements converge on four:

Every version recoverable — the state before any rewrite can be brought back;
Every edit rejectable — model output is a proposal first, and becomes fact only when the user accepts;
Every edit addressable — “change only this section” requires a deterministic answer to “which bytes are this section”;
Review cost bounded — the user must be able to see what changed before deciding.

None of these can be guaranteed by prompting. The rest of this article walks through our implementation layer by layer: the version chain, the proposal ledger, the sectionizer, and the division of diff labor.

The version chain: restore forks, never overwrites

The storage layer is one document_versions table (SQLite, single-user local application):

CREATE TABLE IF NOT EXISTS document_versions (
 id INTEGER PRIMARY KEY AUTOINCREMENT,
 session_id INTEGER NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
 version_no INTEGER NOT NULL,
 content TEXT NOT NULL, -- LaTeX source
 change_summary TEXT,
 parent_version_id INTEGER,
 ...
);

Three design points:

Full snapshot per version, no deltas. content stores the entire source. The cost is storage redundancy; the payoff is that reading any version requires zero replay and both ends of any diff are always materialized. For LaTeX documents in the tens-of-kilobytes range, this trade-off is not close.

version_no is numbered per session. add_version takes MAX(version_no)+1 within the session: the user always sees this document’s v1/v2/v3, never a global v847; the global autoincrement id exists only for foreign keys and lineage. A dedicated test pins this: interleave versions across two sessions, and the second session’s first version is still 1.

parent_version_id records lineage; restore = fork. The restore endpoint in its entirety:

@app.post("/api/versions/{version_id}/restore")
def restore_version(version_id: int, ...):
 """Restore/fork: create a new latest version from a historical
 version's content (parent points at it)."""
 v = db.get_version(version_id)
 ...
 ver = db.add_version(v["session_id"], v["content"],
 f"restored from v{v['version_no']}", v["id"])

Restore deletes and overwrites nothing: it creates a new version from the restored version’s content, with parent pointing at the restored version. The test assertion nails the semantics down: restoring v1 after v2 exists produces v3 — content equals v1’s content, version_no only ever increases, parent_version_id points at v1. “Undoing a restore” is just another restore; every operation is an append.

There is no UPDATE content path anywhere in the backend; manual edits also go through add_version. Immutability is the foundation everything else stands on: diffs get stable endpoints, and proposals get a well-defined base.

The proposal ledger: edits land in the ledger first, the tree only on accept

The first implementation used “plan A”: candidates were not persisted. edit-section computed but did not store; the frontend previewed the diff and called the save endpoint itself — zero new tables, no contact with the version tree. Its limits showed quickly: a page refresh lost the candidate, multiple candidates could not coexist, and the server had no audit trail. Hence the upgrade to “plan B” — the proposed_edits ledger:

CREATE TABLE IF NOT EXISTS proposed_edits (
 session_id INTEGER NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
 base_version_id INTEGER NOT NULL, -- which version the edit is based on
 char_start INTEGER NOT NULL,
 char_end INTEGER NOT NULL,
 old_text TEXT NOT NULL DEFAULT '',
 new_text TEXT NOT NULL DEFAULT '',
 status TEXT NOT NULL DEFAULT 'pending', -- pending/accepted/rejected
 scope TEXT NOT NULL DEFAULT 'section', -- section/whole/bib
 ...
);

Every edit the model produces — section-level, whole-document (scope='whole', range [0, len]), bibliography — first becomes a pending candidate; the version tree is touched only on accept, and accept reuses the same add_version:

flowchart LR
 A[Edit request] --> B{Scope}
 B -->|section / bib| C[edit-section]
 B -->|whole| W[edit-whole]
 C --> P[proposed_edits: pending]
 W --> P
 P -->|reject| X[rejected: version tree untouched]
 P -->|accept| G{base still latest?}
 G -->|no| E[409 candidate stale]
 G -->|yes| S[splice] --> V[add_version: new version]

The accept path carries two 409s: candidate already processed (accepted/rejected cannot be re-processed), and base is stale (base_version_id no longer the latest version). The second is the critical one: a candidate’s character range is computed against its base version; splicing against a base that is no longer the latest would silently drop every change between base and head. An adversarial review had found the same failure on the frontend — editing on an old version and accepting would overwrite subsequent versions — the frontend gained an “edit only on the latest version” gate, and this 409 is the server-side backstop.

Whether every edit should require confirmation got its own round of sourced interaction research. The conclusion first corrected the risk framing: a full rewrite is in fact already revertible (the version chain exists); the real risks of a silent default are scope surprise, cost, and expensive after-the-fact review — not data loss. So the confirmation gate should be light (a banner plus a plan card), not blocking modals everywhere; small section-level edits keep one-click accept. An industry counter-example is on record too: Cursor weakened its diff-approval-style confirmation and the community treated it as a regression (taken from our research notes, not independently verified). The direction matches the HITL literature: confirm only irreversible actions and prefer undo (NN/g); when unsure, narrow the scope of action and disambiguate before acting (Microsoft HAX G10).

The deterministic sectionizer: equal-length masking and dual anchors

Where do a candidate’s char_start/char_end come from? A deterministic sectionizer (~260 lines, zero LLM calls, zero third-party dependencies) parses LaTeX source into a node tree along \section / \subsection / \subsubsection. A node’s editable span runs from its heading command to the next heading whose level is not deeper than its own — editing a subsection replaces only that subsection; editing a parent section takes its children with it. Three design points:

Mask, don’t shift. A \section inside a comment or a verbatim environment (verbatim / lstlisting / minted, …) is a false heading; deleting those regions before parsing would destroy every character offset. The fix replaces them with equal-length whitespace:

def _mask(content: str) -> str:
 """Replace comments and verbatim environments with equal-length spaces
 (all character offsets preserved), so a \section inside them is not
 treated as a real heading — while splice still uses original offsets."""

Headings are located on the masked text; titles and content are read from the original at the same offsets. Parsing and splicing share one coordinate system — no translation step, no translation bugs.

Dual anchors. Each node carries both a positional anchor (n0, n1, … in order of appearance; re-parsing the same content always yields the same result) and a content-fingerprint key (L1:ErrorAnalysis, derived from level + title, with an ordinal suffix when titles repeat). Positional anchors drift when sections are inserted or removed; content keys follow the title and are more stable across versions, so they take precedence during resolution. Abstract and keywords have no \section to address, so pseudo-nodes are constructed from the \abstract{...} / \keyword{...} commands — “fix the abstract” also gets scoped editing instead of collapsing into a full rewrite.

Parse failure degrades, never raises. If no section structure parses, the sectionizer returns editable=False and the caller falls back to whole-document revision. The sectionizer’s job is to narrow scope, not to add a failure point.

The robustness of this layer was largely ground out by adversarial review; three real defects, each now pinned by a regression test: the optional-argument heading form \section[short]{long} was missed (the section merged into its predecessor and was silently lost on accept); \section inside comments and verbatim environments produced ghost nodes (corrupting the splice); and the stale-version overwrite problem described above.

Splice, fragment health checks, and a lesson from the reflection loop

A model-returned section fragment travels back into the master document through three all-code steps: strip_fragment cleanup (remove Markdown fences; if the model disobeyed and returned a full document, cut out the body — the general approach to output cleanup is Part 3 of this series); check_fragment health checks (balanced braces, balanced environments, no \documentclass — soft signals that warn rather than block); and finally a one-line splice:

def splice(content, char_start, char_end, new_text):
 return content[:char_start] + new_text + content[char_end:]

Bytes outside the range are unchanged — that promise is written directly into the test assertions.

One lesson deserves its own paragraph. The autonomous reflection loop’s scoped splicing used positional anchors: if a fragment carried one extra top-level \section, the section count went up by one after splicing, the positional anchors of every subsequent edit in the same round drifted, and the wrong sections got edited — while the regression gate of the time only caught “fewer sections”, not “more”. Silent mis-edits. The fix adds a conservation law to the splice: a fragment must preserve the count of top-level \section commands within the replaced span, otherwise that section is skipped. This too came out of adversarial review, and is now a permanent test.

The division of diff labor: the backend guarantees invariants, line-level rendering goes to the frontend

With versions and candidates in place, “see what changed” was deliberately not built as a backend service:

Layer	Responsibility	What it does not do
`document_versions`	Immutable versions, queryable lineage	No diff computation
`proposed_edits`	`old_text` / `new_text` archived as a pair	No presentation
`split_aligned`	Split a whole-document change per section when skeletons align	Returns `None` on structural change
Frontend `diffLines` (jsdiff)	Line-level highlight rendering	Never mutates data

split_aligned is the only place the backend “computes a diff”, and its conditions are deliberately tight: the top-level \section skeletons of base and new must have the same titles, same order, same count, and the preamble must be byte-identical — only then is the whole-document change split into a list of independently acceptable per-section candidates. When skeletons align, the section ranges are non-overlapping, so accepting one section replaces only that section and leaves everything else byte-for-byte untouched. If any condition fails, it returns None and the change falls back to whole-document all-or-nothing. The preamble check looks strict, but the reason is concrete: if a change lands outside every section, per-section review would miss it.

Line-level diffing goes to jsdiff’s diffLines on the frontend. Display granularity is a UI concern that iterates with the design; the backend promises only testable invariants — immutable versions, bytes-outside-range unchanged, and the safety conditions for splitting.

Quantified: how tests pin these semantics down

Semantics	The test / data that pins it
Restore = fork	After restore: `content` equals the source version, `version_no` only increases, `parent` points at the restored version
`version_no` per session	Interleaved versions across two sessions; the new session’s first version is still 1
Candidate lifecycle	409 on re-processing; 409 on stale base
Sectionizer robustness	29 P0/P1 tests (including 4 added after review); full suite at the time: 173 passed
Per-section review	6 new `split_aligned` tests; full suite at the time: 266 passed
Ownership isolation	Non-owners get 404 on version reads and restore

All of these run with a mocked LLM, fully offline — the data-model layer is precisely the part of an LLM application that conventional assertions can test to death.

Where not to do this

Full snapshots have a ceiling. Storing the whole document per version is right for tens-of-kilobyte LaTeX; at megabyte scale or high save frequency (real-time collaborative editing), switch to delta storage or CRDTs — this model is not designed for that regime.
The ledger is for human-in-the-loop products. If the product is a fully automated pipeline where nobody reviews candidates, the ledger is just an unread queue; what belongs there instead is a deterministic regression gate before anything lands (see Part 11). Likewise, wrapping every low-stakes edit in confirmation is the “crying wolf” anti-pattern NN/g warns about — we keep one-click accept for small section edits and place the confirmation gate only on whole-document changes.
The sectionizer presupposes parseable structure. On unstructured text, editable=False: the version chain still works, but scoped editing degrades to whole-document revision.
split_aligned’s conservatism is a feature, not a defect. When the skeleton changes it refuses to split: structural changes should not be accepted piecemeal — a half-accepted reorganization is more dangerous than a whole-document replacement.
The version tree is not a branching workflow. Lineage is fully recorded, but the presentation is linear with a single tip; exploring several directions in parallel and merging them is unsupported. For git-like branch semantics, this model provides only the foundation.

References

The following sources come from the sourced research notes (devlog) of our interaction study:

Eric Horvitz, Principles of Mixed-Initiative User Interfaces (CHI ‘99) — the principled framework for mixed-initiative interaction.
Microsoft HAX Guidelines — G9 (support efficient undo), G10 (scope services when in doubt / disambiguate before acting), G16 (convey the consequences of actions).
Nielsen Norman Group guidance on confirmation dialogs — confirm only irreversible actions, prefer undo, avoid crying wolf.
Google PAIR, People + AI Guidebook — users resist full autonomy on high-control artifacts; explanation serves understanding.
Cursor community regression feedback on weakened diff approval — taken from our research notes, not independently verified.

Making model output usable: sanitizing and automatic repair

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 3 of the Engineering LLM Applications series. The evidence base is a production-grade AI writing agent that generates LaTeX lab reports; every error message and number in this article comes from its source code and real debugging records.

The problem: the distance between “looks correct” and “usable”

You ask the model to generate a LaTeX lab report. The source it returns is structurally complete, every command spelled right, nothing visibly wrong; you feed it to xelatex and get no legal \end found — even though \end{document} is plainly there at the end of the file. The real cause hides in the header: the model wrote \def\partnerID{% TODO...}, and the % inside the \def line comments out the closing }. A runaway definition swallows everything after \begin{document}. The error location and the cause location are an entire document apart.

This is the most typical gap you hit when starting out with LLM applications: model output and usable artifact are not the same thing. Taking “generate LaTeX → produce a PDF” as the running example, at least three failure layers sit in between:

Won’t compile: syntax looks fine, but it trips corner-case behavior of the engine-and-package combination (the % above; \mathbf below);
Compiles but unusable: compilation “succeeds”, yet the artifact has 0 pages, or figures are missing and every reference renders as [?];
Usable but unsafe: under -shell-escape, one \write18 in the generated body is arbitrary command execution.

The common first reaction is to tighten the prompt or retry the model. Our measured conclusion: prompt constraints lower the incidence but cannot suppress it (the verbatim source comment reads “the model routinely omits these; the prompt can’t hold it down”); retries cost another generation and don’t guarantee convergence. Another counter-intuitive result came from root-causing three classes of generation-quality problems: 2 of the 3 root causes were in our own backend code and template, not in the model. The right place for the fix is a layer of deterministic sanitizing and repair between model output and artifact — zero tokens, unit-testable, predictable behavior. It is also the output-side concretization of the deterministic boundary principle from Part 11 of this series: no fragment enters system state without passing a deterministic gate.

Entry cleanup: strip_fragment and check_fragment

The first gate sits before a fragment enters the document, and handles domain-independent shape problems:

def strip_fragment(text: str) -> str:
 """Clean a model-returned section fragment: strip Markdown code fences;
 if a full document came back by mistake, cut from the first heading to just
 before \end{document} (keeps preamble / top-level end out of the body)."""

Models love wrapping code in Markdown fences; asked to “return only this section”, they may still disobey and return the whole document. The former gets unwrapped; the latter is cut from the first \section-family command to just before \end{document}. The companion check_fragment runs a health check: balanced braces, balanced \begin/\end, no \documentclass, non-empty — all soft signals that warn rather than block, because the check rules themselves can produce false positives.

The pit catalog: deterministic rewriting in sanitize_tex

The second gate runs before compilation and is domain-specific: a pit catalog, every entry earned through a real debugging chain — fixing one exposed the next:

Symptom (real error)	Root cause	Deterministic fix
`no legal \end found` (with `\end{document}` present)	`{%` inside a `\def` line comments out the closing `}`; runaway definition swallows the rest	`{%`→`{}%` on `\def` lines only; the multi-line `\abstract{%` idiom is left alone
`\textfont 11/12 undefined` (`\mathbf`); `Extended mathchar used as mathchar` (`\boldsymbol`)	both conflict with the xelatex + unicode-math combination	rewrite uniformly to unicode-math’s `\symbf`
`Extended mathchar` / `A number should have been here`	bold inside accents (`\hat{\boldsymbol{x}}`)	drop the bold, keep the accent; both nesting orders handled
A string of Undefined errors	template commands like `\makeEngPage` called with their prerequisite variables unset	inject empty defaults after `\begin{document}`
A missing-`\item` / `\noalign` cascade	the two-column table environment instrmlist used as an itemize with `\item`	rewrite `\item Name (Model)` inside the block to `Name & Model \\`; already-correct rows untouched

Two entries deserve expansion.

We fixed the \mathbf entry in the wrong direction once. The first fix was \mathbf→\boldsymbol (the template loads the bm package — seemingly reasonable). Only an end-to-end rerun revealed the direction was wrong: \boldsymbol triggers the same “Extended mathchar used as mathchar” under xelatex + unicode-math. The correct target is unicode-math’s own \symbf. After the correction, recompiling a historical version containing 13 occurrences of \mathbf measured: Extended mathchar 13→0, total errors 48→35. The lesson is twofold: sanitizing rules need empirical verification of their own, or the fix itself introduces regressions; and the rewrite target must be a form that is provably compatible with the current engine combination, not one that is “usually equivalent”.

What value to backfill is a product judgment, not just a technical one. The template-variable backfill carries an exception table: when the English-page author name is missing, the value of the corresponding Chinese variable is copied — author names are shared across both language pages, and that beats an author line reduced to a lone “and”. But the English abstract abstractEng is deliberately left empty: an abstract requires translation, and pushing Chinese text onto the English page would inject wrong information. The red line for automatic repair: backfill only defaults that are provably harmless; never guess semantics.

Artifact and safety: figure placeholders, font substitution, injection stripping

Some problems live not at the syntax layer but at whether the artifact is usable and safe to distribute.

A missing figure takes down the whole document. Models routinely \includegraphics figures that don’t exist in the provided material. A missing figure doesn’t just report File not found — it cascades into a truncated .aux: the whole document renders 0 pages and the PDF won’t open. The fix is a deterministic substitution: any \includegraphics referencing a file absent from the figure directory is replaced with an \fbox placeholder stating “this figure was not provided — supply it or remove the reference”. Zero LLM calls, two error classes eliminated at once, and a readable multi-page PDF comes out.

Platform substitution for a hard-coded font. The template’s cls hard-codes \setmainfont{Times New Roman} (a Windows-bundled font); a Linux container doesn’t have it, producing 100+ font errors per report. The compile step edits only the working-directory copy: on non-Windows platforms the font is replaced with the always-available Latin Modern Roman; the template itself is untouched. The dev machine (WSL2) taught a related lesson: Liberation’s font aliases do not satisfy XeTeX/fontspec’s exact-name matching — installing the real MS fonts took font errors from 6 to 0.

Injection stripping. The template’s minted 2.x forces full -shell-escape (minted.sty:1233 checks \pdf@shellescape=1 and refuses restricted mode), and -shell-escape lets LaTeX run arbitrary shell commands — which makes LLM-generated body text a genuine attack surface. The sanitizing layer strips command-execution primitives from the body: \write18, \ShellEscape, \directlua, and the piped form \input{|cmd}. minted’s own pygmentize invocation lives in the package layer, not the body, so syntax highlighting is unaffected. End-to-end verification: after injecting \immediate\write18{touch ...} and compiling, the target file was not created. The standalone distribution drops -shell-escape entirely — code blocks still typeset via listings, losing only syntax coloring.

After compilation: diagnose and humanize

“Compilation succeeded” lies: under nonstopmode, missing figures and undefined references don’t stop a PDF from being produced. diagnose deterministically tallies four items from the log — error-line count (^!), the list of missing figures, undefined citations, undefined cross-references — so that “succeeded” no longer masks a degraded artifact.

The tally serves two audiences. For the model, extract_errors picks the !-prefixed error lines plus context as input to the compile self-repair loop (Part 6). For the user, humanize maps diagnostics and common errors into “one plain sentence + one clickable action” — along the lines of “2 figures missing: a.png, b.png” with a button “supply the figures, or remove the corresponding \includegraphics”; “Undefined control sequence” becomes “an undefined command was used (possibly a typo or a missing package)”. The raw log is for engineers; it has no place in the product UI.

The pipeline, and the measurements

flowchart TD
 A[Model output] --> B[strip_fragment: unfence / cut fragment from full doc]
 B --> C[check_fragment: balance checks, soft signals]
 C --> D[sanitize_tex: pit-catalog rewrites + exec-primitive stripping]
 D --> E[Assemble workdir: placeholder missing figures; swap font off Windows]
 E --> F[xelatex + bibtex]
 F --> G[diagnose: error count / missing figures / undefined refs]
 G -->|for the user| H[humanize: one sentence + one clickable action]
 G -->|for the model| I[extract_errors → compile self-repair loop]

Measured item	Before	After
`\mathbf`→`\symbf` (recompiling a version with `\mathbf`×13)	Extended mathchar 13, errors 48	0, errors 35
Photonic-crystal auto report (after the first three fixes: `%` in `\def`, math bold, bold-in-accent)	compile failure (runaway)	valid 131KB PDF
Real MS fonts installed on WSL2 (aliases fail exact-name matching)	6 font errors	0
Injected `\immediate\write18{touch ...}`	—	file not created (RCE blocked)

The sanitizing layer is pure functions throughout; when the \symbf fix landed, the full pytest run was 37 passed — every rewrite rule, every backfill exception, and the injection stripping is pinned by an assertion.

Where not to do this

The pit catalog only fixes known pits. Every rule is bound to a specific combination (GPE template + xelatex + unicode-math + minted 2.x); a new template or engine means walking the debugging chain again. It is not a general-purpose LaTeX fixer, and should not try to become one.
Regex rewriting has an inherent collateral-damage surface. Every rule must be narrowed to the lesion: only \def lines, only inside the instrmlist block, already-correct rows preserved verbatim — and pinned by unit tests. One notch too wide and the rule breaks legitimate input; anything that cannot be narrowed to provably-safe should not be auto-fixed, and belongs to the model-driven compile self-repair loop instead.
The ceiling of automatic repair is introducing no wrong information. Deterministically decidable defaults (empty variables, placeholder boxes) may be backfilled; semantic gaps (the English abstract) must be left to a human or the model. Cross that line and repair becomes contamination.
Sanitizing replaces neither upstream constraints nor the downstream loop. The “LaTeX compilability rules” still belong in the system prompt (they lower incidence), and compile self-repair handles the long tail; the sanitizing layer’s role is zero-token interception of high-frequency known pits. The three layers are a union, not substitutes.
Injection stripping is one layer of defense in depth, not all of it. The regex layer does not cover extreme bypasses such as catcode tricks — an acceptable trade-off for a local single-user tool; a multi-tenant server must stack harder layers (a restricted shell-escape baseline, sandboxed execution environments).

Prompt assembly as engineering: layering, contamination, and trimming

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 2 of the Engineering LLM Applications series, grounded — like the rest of the series — in the real source code and tests of a production-grade AI writing agent.

The problem: when the prompt grows too long for anyone to dare touch

When your first LLM feature ships, the prompt is usually one f-string: a role blurb, format requirements, a few examples, the user input, concatenated and sent. That stage is fine — the problems arrive three months later:

Fix one place, break another. The same prompt text is reused by several call paths; wording tuned for scenario A quietly degrades scenario B’s output, and no test turns red.
Example content leaks into output. To teach the model how to write, you paste in a model essay; one day a user finds that essay’s numbers and citations inside their own report.
Multi-turn gets more expensive every turn. Each turn carries the full history, and the history contains entire documents the model previously produced; token spend climbs with turn count, most of it redundant.
Changing a sentence requires a release. The prompt is hard-coded in source; tweaking one line of wording in the field means shipping a new package.

These four failure classes share one root cause: the prompt is treated as a piece of text rather than a build artifact. A build artifact implies three things: a layered structure (each layer with its own change frequency and ownership), incidents (which need incident records), and regression protection (changes are guarded by tests). The rest of this article dissects that practice through prompts.py and its tests, from this series’ evidence-base project (an AI writing agent producing LaTeX lab reports).

Mechanism 1: layered assembly — each layer has its own change frequency

The system prompt is a concatenation of five layers (build_system), not one string:

Layer	Content	Change frequency	Hot-swappable
Role boundary	Identity + hard prohibitions (no AI-speak, no fabricated data or citations)	Very low	Via overlay
Genre spec	Two section skeletons: lab report / course notes	Low	Via overlay
Quality anchors	Writing methodology + golden sample (permanent domain assets)	Low	Via overlay
Writing-judgment skill	Distilled “why we write it this way”	High	Distillable, replaceable
Output-format hard rules	LaTeX template skeleton + 6 compile-error pitfalls	Tracks the template	Via overlay

Three design points:

Move knowledge assets out of code. The methodology (~4KB of Markdown) and the golden sample (a 2.2KB LaTeX technique excerpt) live under the knowledge/ directory as file assets; code only assembles. Together with the bib file and the material pack they form the agent’s domain assets — reviewable and replaceable independently of the code.

Write format rules with their error messages attached. The output-format layer does not just say “do this”; it lists six numbered pitfalls, each with the real compile error that violation triggers — e.g. “before calling \makeEngPage{} you must \def the English metadata … missing any one of them raises Undefined control sequence”, and “bold vectors only with \symbf{}; \mathbf triggers Extended mathchar”. These were baked into the prompt after real failures — the compiler’s error-driven experience, front-loaded.

Trim by call site. Section-level small edits (scoped editing, see The deterministic boundary) assemble with scoped=True: the quality anchors are dropped — methodology plus golden sample come to roughly 3.2k characters, about 2k tokens, and a one-section fragment edit has no use for a full model essay about someone else’s experiment. The role boundary, the distilled judgment layer, and the format rules stay, because the fragment must still be stylistically correct and compilable.

flowchart TD
 A[Assemble system] --> B[Role boundary]
 B --> C[Genre spec: report or notes]
 C --> D{Scoped small edit?}
 D -->|yes| E[Skip quality anchors, keep distilled judgment layer]
 D -->|no, full generation| F{Distilled skill present?}
 F -->|yes| G[Fixed quality anchors, dedup, then judgment layer]
 F -->|no| H[Built-in judgment layer = the anchors]
 E --> I[Output-format hard rules]
 G --> I
 H --> I

Mechanism 2: a real incident — the quality anchors got skipped

Layering is not about tidiness on paper; it exists because not layering caused a real incident. The incident record sits in the docstring of build_system itself (translated; the source is Chinese):

"""⚠️ An early L1 refactor made the "distilled template" path (skill_prompt
non-empty) skip builtin_skill entirely → golden sample + methodology lost →
generation depth/style collapse (the root cause of "output is worse with
distilled templates"). Quality anchors are now extracted and always injected,
with dedup for old built-in/legacy templates that carry their own anchors."""

The post-mortem: in the early implementation the quality anchors were bundled inside the “built-in skill”; a refactor made the “use a distilled template” path skip the built-in skill entirely — golden sample and methodology vanished together, and generation depth and style collapsed. The symptom read as “distilled templates produce worse output” and was for a while investigated as a distillation-quality problem; the actual cause was a missing layer in assembly.

The fix hardened into two structural decisions. First, decouple quality anchors from the skill layer: the anchors are “permanent domain assets”, independent of whether a distilled template is in use; a distilled template carries only the judgment layer (why to write it this way) and is not allowed to displace the depth/style anchors. Second, dedup for legacy templates — old built-in templates that already carry the anchors are not injected twice:

if skill:
 anchors = _quality_anchors(kind)
 if anchors and "金样例" not in skill: # skip if the skill already embeds the golden sample
 parts.append(anchors)
 parts.append(skill)

The transferable lesson: layered assembly must state explicitly which layers are unconditionally present — otherwise any “looks equivalent” refactor can silently remove a layer. A prompt missing a layer throws no error; it only degrades.

Mechanism 3: few-shot contamination guards

The golden sample is a double-edged sword. It is a model-essay excerpt about YBCO — someone else’s experiment — used to teach “how to write with depth, to spec, and compilably”. The natural risk of few-shot is content contamination: the model copies the sample’s subject, numbers, and citations into the user’s report — fabricating data by way of the example. There are three guards, all at the wording level:

An isolation label at injection time. The assembly code wraps the golden sample in a declaration: “it is a model essay about another experiment — never copy its subject / title / keywords / numbers / citations / section layout; you must write strictly from this session’s materials, about the experiment you actually did.”
The sample file carries its own immunization comments. fewshot_ybco.tex is not a complete report but a 2.2KB technique excerpt: only an equation-bearing principle paragraph, a booktabs table, and similar craft demonstrations survive. Its header comment restates “the topic below is only an example … never copy this excerpt’s title, data, citations, or sections”, and even the table caption reads “example — a real report must contain this experiment’s real data”.
The methodology layer adds an iron rule. The methodology file’s own hard-rules section independently repeats: “never copy the model essay’s / golden sample’s subject, numbers, or citations — that is someone else’s experiment.”

The triple redundancy is deliberate: wording-level protection is probabilistic, and a single declaration can be diluted in a long context. The final backstop is not in the prompt layer at all — a provenance check runs server-side against the full materials, deterministically. That belongs to the deterministic-boundary side of the system.

Mechanism 4: multi-turn history folding and material trimming

In multi-turn revision, what the model needs is the thread of per-turn feedback (what the user objected to first, what changed next) — not the full text of every historical version. Yet the assistant’s history messages are precisely entire LaTeX documents. The folding rule in build_messages:

if role == "assistant" and "\\documentclass" in content:
 content = "[此前已生成的文档版本，此处省略源码]" # "earlier document version, source elided"
elif len(content) > max_chars:
 content = content[:max_chars] + "…（略）"

Three points. Full documents are replaced by a one-line placeholder — the current document is already supplied separately in this turn’s user content, so re-sending every historical version burns tokens for zero value. Non-document assistant notes are kept — e.g. an auto-exploration conclusion like “the data is in the xrd/ directory” is a fact later turns need. Everything else over 4,000 characters is truncated, and only the last 40 history messages are kept. Dedicated tests pin this behavior (document folding, note preservation, count capping).

Materials get the same deterministic trimming (zero LLM calls, zero tokens): in revision turns, materials over 12k characters are filtered by lexical overlap with the current feedback; pure-number matrix lines (bare CSV) are always dropped — the figure pipeline has already digested the raw data, and prose writing has no use for row-level numbers. If filtering keeps fewer than two lines and under 40 characters, it falls back to the head of the material — filtering must never destroy all context. On the chat path, materials become a per-file-quota stable digest (each file guaranteed at least a 600-character allowance, so later files are not starved by earlier large ones), depending only on the material itself and therefore byte-identical across turns — usable as a stable first-message prefix that hits DeepSeek’s automatic prefix caching.

Mechanism 5: the overlay hot-swap layer and its regression gates

The last cost of hard-coded prompts is release coupling: changing one sentence in the field means shipping a new installer. The remedy is the agent-pack overlay:

def _prompt_part(rel: str, default: str) -> str:
 return _overlay_read(rel) or default

Role, genre specs, format rules, methodology, and golden sample are all overridable: a same-named non-empty file in the overlay directory wins; otherwise the built-in default applies. _overlay_read reads on every call (lazy loading) — swap the pack and the next generation uses it, with no recompile and no restart; a failed read or an empty file returns the empty string and falls back to the built-in, so an empty pack can never wipe out the knowledge.

The danger of a hot-swap layer is that behavior with no overlay must not change by a single byte. Two groups of offline tests guard this:

Overlay behavior tests: no overlay → bundled defaults; overlay takes precedence and is lazy (change the content from V1 to V2 and the next call sees V2); an empty file falls back to the built-in.
Prompt-shape assertions in the golden regression gate: the scoped prompt must still contain the “rewrite only this block” instruction and the prohibition wording (with \documentclass and \end{document} appearing inside the prohibition); build_user(gaps=None) must be byte-identical to calling without gaps — proving a newly added parameter did not contaminate the existing path.

Both groups are plain pytest, fully offline: whoever edits the prompt back to “output the full document”, or lets the default path drift, turns CI red immediately.

Costs and gains

Mechanism	Cost of not doing it	Effect in practice
Layering + scoped trimming	Full system prompt on every small edit	~2k tokens of quality anchors saved per scoped edit; with scoped editing, output tokens drop an order of magnitude
Fixed anchor injection	Relying on each template to carry its own	Fixed the “distilled-template collapse” incident; dedup prevents double injection
Contamination labels × 3	Bare few-shot, model-essay content leaking	Three wording-level guards + server-side provenance backstop
History folding + material trimming	Re-sending all history and materials every turn	Full documents fold to a one-line placeholder; 40-message / 4,000-character double cap; stable prefix hits caching
Overlay hot-swap	Every wording change is a release	Swap the pack, next generation applies it; no-overlay behavior byte-identical, guarded by golden gates

Where not to do this

Do not layer during prototyping. With one call path and a prompt still changing daily, one f-string is the correct form; layering and golden gates are for the stage where the prompt has stabilized, multiple call sites reuse it, and the field needs hot swaps. Premature abstraction pays every cost twice.
Shape assertions do not test semantics. The golden gate asserts keyword presence and byte identity; a rewording that preserves the keywords passes green while quality shifts. Semantic drift needs online evaluation sets — see part 8 of this series, The testing pyramid for LLM applications.
Contamination guards are probabilistic. Isolation labels lower the copy probability; they do not zero it. High-risk domains (numbers, citations) need deterministic verification outside the prompt as the backstop — the defense cannot live in wording alone.
Hot swapping cuts both ways. The overlay lets a field pack bypass the bundled tests — golden gates guard the built-in defaults, not the pack’s contents. Hot-swap capability must come with an acceptance process for the pack itself; otherwise it is an unguarded change channel.

The deterministic boundary: a first principle for LLM application architecture

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 11 of the Engineering LLM Applications series — where the mechanisms of the preceding ten articles converge into one principle. The series is grounded in the implementation and measurements of a production-grade AI writing agent (FastAPI + React + Tauri, 351 test cases).

The problem: probabilistic components, engineering obligations

Once an LLM is embedded in a production application, the hard engineering problem is not that the model is insufficiently smart. It is that a probabilistic component degrades system properties in several predictable ways:

Scope escape. Ask the model to “fix one section” and it rewrites the whole document. Empirical measurement of full-rewrite revision (arXiv:2601.13217) shows revisions regress 16–27% of already-covered content on average — not a model slip, but a statistical property of open-ended rewriting.
Loop collapse. In long contexts, agents degenerate into calling the same tool with identical arguments. A public case (qwen-code issue #4695) records 43 consecutive git status calls consuming 8.9M tokens in one session; SDK-level retries cannot help, because every call succeeds at the API layer.
Cost drift. If every turn carries the full context, a “fix the keywords” request pays for the entire document plus all source material — we measured roughly 6–7k input tokens for such requests, of which about 1.5k carries information.
Untestable regressions. Non-deterministic output defeats conventional assertions; a prompt edit silently shifts quality and CI has nothing to catch it with.

Against these failure modes, our project converged on one architectural principle, which this article calls the deterministic boundary:

Never ask the model a question deterministic code can answer; let the model produce only the minimal necessary fragment; admit no fragment into system state without passing a deterministic gate.

The principle is orthogonal to two existing families of tooling: structured outputs / function calling address format determinism (what the output looks like); guardrail frameworks address content compliance (what the output may say). The deterministic boundary addresses scope and state determinism — which part of the system the output is allowed to modify, and under what conditions it lands. The str_replace / fast-apply pattern in code editors (model produces the diff, application is deterministic) is the same principle expressed in another domain.

The rest of this article walks through five mechanisms, using document revision as the running example. The stack is Python/FastAPI and the document format is LaTeX, but the mechanisms are format-agnostic: they apply to any domain with parseable structure.

Mechanism 1: deterministic targeting — the sectionizer

Scoped editing presupposes a deterministic answer to “what is the character range of the target section.” Our sectionizer (~260 lines, zero LLM calls, zero dependencies) parses LaTeX source into a node tree along \section / \subsection / \subsubsection, assigning each node a char_start / char_end. Two design points deserve expansion:

Mask, don’t shift. A \section inside a comment or a verbatim environment is a false heading — but deleting those regions before parsing would invalidate every character offset. The solution is to replace them with equal-length whitespace:

def _mask(content: str) -> str:
 """Replace comments and verbatim environments with equal-length spaces
 (all character offsets preserved), so a \section inside them is not
 treated as a real heading — while splice still uses original offsets."""

Headings are located on the masked text; content is read from the original at the same offsets. Parsing and splicing share one coordinate system, so there is no translation step — and no translation bugs.

Dual anchors. Every node carries both a positional anchor (n0, n1, stable within one version) and a content-fingerprint key (L1:ErrorAnalysis, derived from level + title). Positional anchors drift when sections are inserted or removed; content keys follow the title and survive across versions, so they take precedence during resolution. This is the cheapest possible mechanism for “refer to the same section across versions”: no diff algorithm, just two string keys.

The failure mode is deterministic too: if no structure parses, the sectionizer returns editable=False and the caller falls back to whole-document revision. The boundary machinery never raises into the main flow — its job is to narrow the model’s scope, not to add a failure point.

Mechanism 2: scoped generation and deterministic splicing

After targeting, the prompt sent to the model contains only the target section (plus hard rules against scope escape), and the model returns only a replacement fragment for that section. The fragment’s path back into the master document is entirely code:

strip_fragment — cleanup: remove Markdown fences; if the model disobeyed and returned a full document, cut out the span from the first heading to just before \end{document};
check_fragment — health check: balanced braces, balanced environments, no \documentclass (soft signals that warn rather than block);
splice(content, start, end, new_text) — a one-line reassembly in which everything outside the range is byte-identical.

Byte-identical is a testable promise, not an adverb: the assertion is written directly into the test suite. One implementation lesson: a fragment must preserve the count of top-level \section commands within the replaced span, otherwise the positional anchors of subsequent edits in the same round drift and hit the wrong section. That bug was caught by an internal adversarial review and is now pinned by a permanent test.

Mechanism 3: zero-token pre-routing

“Which part does this feedback want to change” is a routing problem. The default is to ask a model, but a substantial share of feedback carries an unambiguous explicit target: “the bibliography format is wrong”, “the abstract is too long”, “the units in Table 2 are off”, “rewrite §3.1”. Those are decided by rules:

signal classes = { bibliography terms, abstract/keywords, section references
 (Table N / Figure N / § numbers / unique section titles) }
exactly one class hit → route directly (0 tokens)
no signal / conflicting classes / "whole document" wording → fall back to LLM routing

The design stance is conservative-first: prefer falling back to the LLM over ever guessing. Disambiguation is deterministic as well — when both “Data Processing” and “Data Processing and Results” match, the longer title wins. Combined with the narrow prompt from Mechanism 2, this cut input tokens for keyword-fix-type requests from roughly 6–7k to about 1.5k in our measurements; routing itself costs nothing, adds no latency, needs no API key, and runs at full speed in offline tests.

Mechanism 4: the zero-token regression gate

The autonomous agent’s reflection loop (draft → self-revise × N) is where scope escape does the most damage. Beyond constraining revision into a structured edit plan (schema-enforced changes[], where an empty array means converged and terminates early) plus scoped splicing, there is one final gate before anything lands — regression_check, six fully deterministic criteria:

loss of the compilable skeleton (\documentclass / \end{document});
length collapse below 60% of the previous version (truncation, or “rest unchanged”-style elision);
structural score decrease (eight equal-weight checks: section count, error analysis, data tables, equations, no TODO placeholders, …);
shrinkage of the \cite key set;
fewer sections;
fewer figures or tables.

Any trigger reverts to the previous version and stops the iteration. The reason for choosing deterministic criteria over one more LLM judge call is stated in the module’s own comments: this particular failure mode — losing content — is deterministically detectable, whereas the pairwise judge’s validity in this domain is unverified and would cost an extra call per round. More correct, and cheaper.

The gate has its own edge case: bibliography generation runs after reflection, so with an empty bib the “citations closed” check is false for any draft containing \cite, and the gate would misread “added a citation” as degradation. The fix excludes that check from the gate score while the bib is empty; citation loss is still caught separately by the key-set-shrinkage criterion. A gate is code, and code has bugs — but a gate’s bugs can be pinned by unit tests, while model drift cannot. That asymmetry is precisely why the gate belongs on the deterministic side.

Mechanism 5: putting the boundary itself under CI — prompt-shape gates

Offline tests cannot run a real LLM, so they cannot measure generation quality — but they can test the shape of a prompt: whether the scoped prompt still demands “change only this section”; whether the assembly function’s output on gap-free input is byte-identical to a golden file. If someone edits the prompt back to “output the full revised document”, plain pytest goes red. The most fragile part of the deterministic boundary — constraints written in natural language — thereby gets the same regression protection as code. A companion gate validates the scorer itself: gold-standard good reports must score ≥ 0.75, bad ones ≤ 0.35, with a gap ≥ 0.4 — the scorer must demonstrably encode quality before it is allowed to act as a gate.

The whole picture, and the measured gains

flowchart TD
 A[Edit request] --> B{Deterministic pre-router}
 B -->|explicit target hit| S[Scoped prompt: target section only]
 B -->|no signal / conflict| L[LLM router] --> S
 S --> G[LLM generates section fragment]
 G --> C[Deterministic cleanup and health check]
 C --> P[Splice: bytes outside range unchanged]
 P --> H{Who is in the loop}
 H -->|conversational editing| U[Proposal ledger: lands only on user accept]
 H -->|autonomous reflection| R{Zero-token regression gate}
 R -->|pass| V[Commit as new version]
 R -->|degraded| X[Revert to previous version, stop]

Mechanism	Alternative	Measured gain
Sectionizer + splice	Full-document rewrite by the model	Bytes outside the target range unchanged; avoids the 16–27% content-regression mode
Zero-token pre-router	LLM routing on every request	0-token decisions for explicit targets; input 6–7k → ~1.5k tokens
Regression gate	Accept every revision / LLM judge	One judge call saved per round; degradation reverts, convergence stops early
Prompt-shape gate	Manual review of prompt edits	The deterministic share of 351 tests runs fully offline, in CI

Where the principle does not apply

In keeping with this series’ convention, the honest boundary of the boundary:

It presupposes parseable structure. LaTeX has \section, code has ASTs, Markdown has headings; free-form prose has none. Without structural anchors there is no sectionizer — the boundary collapses to just cleanup and gates.
The boundary is code, and code has carrying costs. The sectionizer must handle starred headings, optional short-title arguments, comment and verbatim masking; we maintain dozens of unit tests for this layer alone. If the product’s shape is still changing fast, building the boundary early makes every change cost twice.
It protects stock, not flow. The boundary’s value grows with the value of existing assets: it prevents version N from being wrecked, and contributes nothing to generating version 1. Tasks that are creation-dominated, with no existing asset to protect, do not need it.
Deterministic criteria only measure the measurable. The regression gate prevents getting worse; it cannot certify getting better. Preference-level quality still requires pairwise judges or humans — which is the subject of another article in this series (validity engineering for LLM-as-judge).

References

Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (arXiv:2601.13217) — empirical measurement of the 16–27% content-regression mode in full-rewrite revision; structured edit plans and scoped revision are the remedies it names.
PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing (arXiv:2604.05018) — structured multi-agent pipelines beating autonomous baselines (+50–68% / +14–38%); its accept-or-revert refinement gate reports 79–81% win rate with 0% degradation (body-text detail, not independently verified).
Self-Refine: Iterative Refinement with Self-Feedback (arXiv:2303.17651) — feedback must be specific and actionable; generic feedback measurably hurts, and tasks where errors are hard to self-detect need external signals.
qwen-code issue #4695 — a public case of tool-loop collapse (43 identical calls / 8.9M tokens).

The token economics of LLM applications

Wed, 29 Jul 2026 00:00:00 +0000

Also in 中文. Part 1 of the Engineering LLM Applications series. The series is grounded in the implementation and measurements of a production-grade AI writing agent (FastAPI + React + Tauri), and discusses transferable engineering methods.

The problem: why does fixing one keyword cost 6–7k tokens

When you build an LLM application, the first constraint that forces an architectural decision is usually not model capability — it is the bill. Our project (a LaTeX lab-report writing agent) received a blunt question from a user: “why did you send 6k?” — he only wanted to change the report’s keywords.

The precise ledger (before the fix):

Component	Content	Size (approx.)
System prompt	Role rules + report spec + writing methodology (1,948 chars) + golden-sample exemplar (1,235 chars) + LaTeX template rules	~3k tokens
Session material	First 6,000 chars of `material` — including the entire experiment txt bundle (data matrices, field-sweep parameters, …)	~3–4k tokens
Target block + instruction	One `\keyword{...}` line + the edit request	~0.1k
Total		~6–7k in / ~50 out

The root cause is not the model but one-size-fits-all context assembly: whatever the edit, the full set of quality anchors plus the full material gets stuffed in. Fixing a keyword has no use for a complete exemplar report about a different experiment, and even less for raw data matrices. The user’s follow-up became the design charter for this pipeline: “the process may be complex, but the goal is to save tokens.” Cost optimization is not cutting features; it is layering context by problem type — minimal context per type. Below are the four layers we shipped.

Layer 1: zero-token deterministic pre-routing

“Which part does this feedback want to change” is a routing problem, and the default is to ask a model. But a substantial share of feedback carries an unambiguous explicit target: bibliography, abstract, keywords, “Table 2”, “Figure 3”, “§3.1”, a uniquely matching section title. These are decided with regexes and string matching (_fast_route — zero LLM calls, no API key even needed): exactly one signal class hit → route directly; “whole document” wording, no signal, or conflicting classes → fall back conservatively to LLM routing, never guess.

The gain is more than one saved routing call: the decision has zero latency, is 100% explainable (the log is tagged “deterministic fast decision, 0 tokens”), and offline tests run at full speed. This pattern is developed further as part of the deterministic-boundary principle in Part 11.

Layer 2: tiered system prompts

The most expensive part of the system prompt is the quality anchors: the writing methodology plus the golden-sample exemplar, about 3.2k characters, roughly 2k tokens. They are indispensable for whole-document generation — an early refactor once dropped them and generation depth and style collapsed immediately — but useless for small edits like “fix one section” or “fix the keywords”.

So build_system(scoped=True) adds a tier: paragraph-level and metadata edits drop the quality anchors while keeping the role rules (no fabrication), the format spec, and the distilled judgment layer — the fragment must still be stylistically correct and compilable. Whole-document generation does not take this tier; its anchors stay. Each scoped call saves about 2k tokens. The full design of layered prompt assembly is the subject of Part 2.

Layer 3: material trimming and stable prefixes

The material layer does two things, and the second matters more than the first.

Deterministic trimming. select_material filters at zero token cost: pure-number matrix lines (bare CSV) are dropped; lines with lexical overlap with the target section or the feedback, or “data lines with textual annotation” (e.g. “measured 0.5 mm”), are kept, capped at 2,000 characters. The filter has a floor — it never filters the context away entirely:

# keep >=2 lines or >=40 chars; otherwise treat filtering as failed
# and fall back to the first `cap` chars
return out if (len(keep) >= 2 or len(out) >= 40) else m[:cap]

The whole-document revision path gets the same treatment (triggered only above 12k characters; never on first-version generation): autonomous exploration used to write up to 120k characters of raw files back into the material, after which every chat / whole-edit turn carried 40–70k tokens — even when the feedback was just “make the tone more formal”. After filtering: 8–15k per turn.

Stable prefixes. A deep audit produced an honest conclusion: on providers with prefix caching, a slimmer system prompt saves roughly nothing in direct cost (its value is focus — keeping irrelevant text from interfering); the real money is in the material block that misses the cache every turn. DeepSeek’s context cache is fully automatic, and the hit price is about 0.8% of a miss (v4-pro: hit $0.003625/M vs miss $0.435/M — vendor list prices verified online, 2026-07). To collect that discount, the material message must be byte-identical across turns — so the trimming and per-file digest functions depend only on the material itself, never on the current turn’s feedback, and the result is pinned as a fixed messages[0] prefix. Measured input cost: about -49%, multiplicative with the relevance filtering above.

The Anthropic path requires declaring the cache explicitly, and the TTL is a genuine trade-off:

system=[{"type": "text", "text": system,
 "cache_control": {"type": "ephemeral", "ttl": "1h"}}],

The reasoning is written in the source comment: users revise documents at human pace, so a 5-minute TTL expires between turns — you pay the cache-write premium every turn and never hit. The 1h tier writes at 2× price, reads at 0.1×, and every hit renews the TTL, matching the human rhythm. Below the model’s minimum prefix threshold it silently skips caching — harmless. Measured: Claude-path input cost down 54–65%.

Layer 4: per-step model tiering

Pipeline steps are not uniformly hard: the exploration tool loop (reading files, deciding what to read next), the material health check (JSON classification), BibTeX generation, and the English-abstract translation are simple, high-frequency tasks; drafting and reflective revision are what need the flagship model. So the model is passed per step: simple steps go to the cheap tier (DeepSeek V4 Flash — input $0.14/M, output $0.28/M, about 1/3 of Pro’s unit price; the vendor claims Flash matches Pro on simple agent tasks at roughly 12× lower cost — vendor claim, not independently verified), while drafting, reflection, figure generation, and compile self-repair stay on Pro. With no cheap tier configured, everything falls back to the main model — zero behavior change; and cost accounting accrues per actual model of each call, because a single flat rate misprices runs that mix the two tiers.

The measured ledger

Typical inputs after layering context by problem type:

Problem type	System prompt	Material	Typical in
Routing (explicit target)	—	—	0 (deterministic decision)
Routing (everything else)	Small router system	none	~1.5–2.5k
Edit abstract/keywords	Scoped tier	none (document outline digest instead)	~1.5k
Edit one body section	Scoped tier	relevance-filtered, ≤2,000 chars	~2–3k
Whole-document rewrite	Full (quality anchors intact)	full	10k+ (money well spent)

The opening example: ~6–7k input tokens before → 0 (fast route hits the keywords class) + ~1.5k after. The whole-document revision path drops from 40–70k to 8–15k per turn, then multiplies with the ~-49% prefix-cache discount. Compile self-repair was also switched to a fresh minimal context (current draft + error excerpt + template rules), saving 30–50k tokens per repair round.

Observation ships with the optimization: every call collects real token usage via usage_out (including cache-hit counts) into the process log and trace, so agent-lab can show “which step spent how much”. This round of changes added 10 backend tests (backend 290 passed, agent-lab 15 passed). One negative lesson: the accounting was once wrong — cost estimation did not discount cache hits, so autonomous-run costs displayed up to several times too high. Fix the observability first, or the optimization aims at the wrong target.

When not to do this

Never filter material for first-version generation. The material is the sole source of facts — err on the side of completeness; relevance filtering is for revision turns only.
Never slim the system prompt for whole-document generation. The quality anchors are where depth and style come from; that spend is worth it.
Never economize on the verification layer. Provenance checking still runs against the complete material server-side — you save on what is fed to the model, not on validation.
Nominal savings ≠ real savings. Once caching hits, many “big” optimizations shrink: splitting the exploration/figure contexts looked like 50k+ tokens saved, but the cache already absorbed about 90% of it, leaving 10–20% real savings — we downgraded that item accordingly. Rank optimizations by post-cache-discount numbers.
Fix observation before optimizing. When accounting ignores cache hits, the most expensive path looks even more expensive than it is, and steers the optimization effort in the wrong direction.