Making model output usable: sanitizing and automatic repair

Also in 中文. Part 3 of the Engineering LLM Applications series. The evidence base is a production-grade AI writing agent that generates LaTeX lab reports; every error message and number in this article comes from its source code and real debugging records.

The problem: the distance between “looks correct” and “usable”

You ask the model to generate a LaTeX lab report. The source it returns is structurally complete, every command spelled right, nothing visibly wrong; you feed it to xelatex and get no legal \end found — even though \end{document} is plainly there at the end of the file. The real cause hides in the header: the model wrote \def\partnerID{% TODO...}, and the % inside the \def line comments out the closing }. A runaway definition swallows everything after \begin{document}. The error location and the cause location are an entire document apart.

This is the most typical gap you hit when starting out with LLM applications: model output and usable artifact are not the same thing. Taking “generate LaTeX → produce a PDF” as the running example, at least three failure layers sit in between:

1. Won’t compile: syntax looks fine, but it trips corner-case behavior of the engine-and-package combination (the % above; \mathbf below);
2. Compiles but unusable: compilation “succeeds”, yet the artifact has 0 pages, or figures are missing and every reference renders as [?];
3. Usable but unsafe: under -shell-escape, one \write18 in the generated body is arbitrary command execution.

The common first reaction is to tighten the prompt or retry the model. Our measured conclusion: prompt constraints lower the incidence but cannot suppress it (the verbatim source comment reads “the model routinely omits these; the prompt can’t hold it down”); retries cost another generation and don’t guarantee convergence. Another counter-intuitive result came from root-causing three classes of generation-quality problems: 2 of the 3 root causes were in our own backend code and template, not in the model. The right place for the fix is a layer of deterministic sanitizing and repair between model output and artifact — zero tokens, unit-testable, predictable behavior. It is also the output-side concretization of the deterministic boundary principle from Part 11 of this series: no fragment enters system state without passing a deterministic gate.

Entry cleanup: strip_fragment and check_fragment

The first gate sits before a fragment enters the document, and handles domain-independent shape problems:

def strip_fragment(text: str) -> str:
    """Clean a model-returned section fragment: strip Markdown code fences;
    if a full document came back by mistake, cut from the first heading to just
    before \end{document} (keeps preamble / top-level end out of the body)."""

Models love wrapping code in Markdown fences; asked to “return only this section”, they may still disobey and return the whole document. The former gets unwrapped; the latter is cut from the first \section-family command to just before \end{document}. The companion check_fragment runs a health check: balanced braces, balanced \begin/\end, no \documentclass, non-empty — all soft signals that warn rather than block, because the check rules themselves can produce false positives.

The pit catalog: deterministic rewriting in sanitize_tex

The second gate runs before compilation and is domain-specific: a pit catalog, every entry earned through a real debugging chain — fixing one exposed the next:

Two entries deserve expansion.

We fixed the \mathbf entry in the wrong direction once. The first fix was \mathbf→\boldsymbol (the template loads the bm package — seemingly reasonable). Only an end-to-end rerun revealed the direction was wrong: \boldsymbol triggers the same “Extended mathchar used as mathchar” under xelatex + unicode-math. The correct target is unicode-math’s own \symbf. After the correction, recompiling a historical version containing 13 occurrences of \mathbf measured: Extended mathchar 13→0, total errors 48→35. The lesson is twofold: sanitizing rules need empirical verification of their own, or the fix itself introduces regressions; and the rewrite target must be a form that is provably compatible with the current engine combination, not one that is “usually equivalent”.

What value to backfill is a product judgment, not just a technical one. The template-variable backfill carries an exception table: when the English-page author name is missing, the value of the corresponding Chinese variable is copied — author names are shared across both language pages, and that beats an author line reduced to a lone “and”. But the English abstract abstractEng is deliberately left empty: an abstract requires translation, and pushing Chinese text onto the English page would inject wrong information. The red line for automatic repair: backfill only defaults that are provably harmless; never guess semantics.

Artifact and safety: figure placeholders, font substitution, injection stripping

Some problems live not at the syntax layer but at whether the artifact is usable and safe to distribute.

A missing figure takes down the whole document. Models routinely \includegraphics figures that don’t exist in the provided material. A missing figure doesn’t just report File not found — it cascades into a truncated .aux: the whole document renders 0 pages and the PDF won’t open. The fix is a deterministic substitution: any \includegraphics referencing a file absent from the figure directory is replaced with an \fbox placeholder stating “this figure was not provided — supply it or remove the reference”. Zero LLM calls, two error classes eliminated at once, and a readable multi-page PDF comes out.

Platform substitution for a hard-coded font. The template’s cls hard-codes \setmainfont{Times New Roman} (a Windows-bundled font); a Linux container doesn’t have it, producing 100+ font errors per report. The compile step edits only the working-directory copy: on non-Windows platforms the font is replaced with the always-available Latin Modern Roman; the template itself is untouched. The dev machine (WSL2) taught a related lesson: Liberation’s font aliases do not satisfy XeTeX/fontspec’s exact-name matching — installing the real MS fonts took font errors from 6 to 0.

Injection stripping. The template’s minted 2.x forces full -shell-escape (minted.sty:1233 checks \pdf@shellescape=1 and refuses restricted mode), and -shell-escape lets LaTeX run arbitrary shell commands — which makes LLM-generated body text a genuine attack surface. The sanitizing layer strips command-execution primitives from the body: \write18, \ShellEscape, \directlua, and the piped form \input{|cmd}. minted’s own pygmentize invocation lives in the package layer, not the body, so syntax highlighting is unaffected. End-to-end verification: after injecting \immediate\write18{touch ...} and compiling, the target file was not created. The standalone distribution drops -shell-escape entirely — code blocks still typeset via listings, losing only syntax coloring.

After compilation: diagnose and humanize

“Compilation succeeded” lies: under nonstopmode, missing figures and undefined references don’t stop a PDF from being produced. diagnose deterministically tallies four items from the log — error-line count (^!), the list of missing figures, undefined citations, undefined cross-references — so that “succeeded” no longer masks a degraded artifact.

The tally serves two audiences. For the model, extract_errors picks the !-prefixed error lines plus context as input to the compile self-repair loop (Part 6). For the user, humanize maps diagnostics and common errors into “one plain sentence + one clickable action” — along the lines of “2 figures missing: a.png, b.png” with a button “supply the figures, or remove the corresponding \includegraphics”; “Undefined control sequence” becomes “an undefined command was used (possibly a typo or a missing package)”. The raw log is for engineers; it has no place in the product UI.

The pipeline, and the measurements

The sanitizing layer is pure functions throughout; when the \symbf fix landed, the full pytest run was 37 passed — every rewrite rule, every backfill exception, and the injection stripping is pinned by an assertion.

Where not to do this

1. The pit catalog only fixes known pits. Every rule is bound to a specific combination (GPE template + xelatex + unicode-math + minted 2.x); a new template or engine means walking the debugging chain again. It is not a general-purpose LaTeX fixer, and should not try to become one.
2. Regex rewriting has an inherent collateral-damage surface. Every rule must be narrowed to the lesion: only \def lines, only inside the instrmlist block, already-correct rows preserved verbatim — and pinned by unit tests. One notch too wide and the rule breaks legitimate input; anything that cannot be narrowed to provably-safe should not be auto-fixed, and belongs to the model-driven compile self-repair loop instead.
3. The ceiling of automatic repair is introducing no wrong information. Deterministically decidable defaults (empty variables, placeholder boxes) may be backfilled; semantic gaps (the English abstract) must be left to a human or the model. Cross that line and repair becomes contamination.
4. Sanitizing replaces neither upstream constraints nor the downstream loop. The “LaTeX compilability rules” still belong in the system prompt (they lower incidence), and compile self-repair handles the long tail; the sanitizing layer’s role is zero-token interception of high-frequency known pits. The three layers are a union, not substitutes.
5. Injection stripping is one layer of defense in depth, not all of it. The regex layer does not cover extreme bypasses such as catcode tricks — an acceptable trade-off for a local single-user tool; a multi-tenant server must stack harder layers (a restricted shell-escape baseline, sandboxed execution environments).