Wiring external signals into the loop: compile-and-repair

Also in 中文. Part 6 of the Engineering LLM Applications series.

The problem: models cannot fix errors they cannot see

Scenario: an autonomous agent generates a LaTeX lab report, and xelatex fails. Who fixes it?

The reflex is to ask the model to “check it again.” The evidence against that path is specific. One of Self-Refine’s conclusions (arXiv:2303.17651) is that on tasks where errors are hard to self-detect, pure self-reflection is nearly useless, and gains only return once an external verification signal is wired in. The systematic review in When Can LLMs Actually Correct Their Own Mistakes (TACL 2024) is blunter: prompting-only self-critique does not improve — and can degrade — arithmetic and code tasks, and the flattering self-correction numbers in the literature mostly leaned on an oracle (stopping only when the known-correct answer appears). Compile errors are the canonical hard-to-self-detect case: the model rereads the \def line it just wrote and cannot see that a % inside the braces comments out the closing brace — but xelatex sees it, and its error report cannot be faked.

So the right question is not “how do we make the model self-check” but “how do we wire an objective external verifier into the generation loop.” This article dissects the compile-and-repair loop we shipped in an AI writing agent — really compile after generation; on failure, extract the errors and feed them back for a repair round — and the engineering constraints that keep this deceptively simple loop under control in production.

From patching to really compiling

Before the compiler was wired in, the system already had a deterministic sanitizer layer (part 3 of this series): \mathbf→\symbf, % comments inside \def lines, backfilling template variables the model forgot to set — the known pitfalls that rules can encode. But patches cannot enumerate the long tail. The devlog states the goal of this step plainly: “not patching with the sanitizer, but having the agent really compile after generating the report, feed the xelatex errors back for a round of its own fixes on failure, and push the first-pass success rate to the max.”

That fixed the division of labor: rules handle the known pitfalls (zero tokens); the compiler backstops the long tail (one LLM call per round). In practice the sanitizer has already cured the common traps, so the first compile check usually passes outright; the repair branch only catches what remains. The loop itself is three steps:

1. compile_tex — assemble a complete working directory (article.tex + cls + bib + figures), run xelatex → bibtex → xelatex ×2, return {ok, pdf_path, log};
2. extract_errors — pull from the log the lines starting with !, the l. line-number lines, and Runaway / Emergency stop lines, capped at 1800 characters — never feed back the whole log;
3. on failure, feed the error excerpt back to the model, demand the full revised source, at most 2 rounds.

The cap of 2 is not arbitrary: a survey finding (How Many Tries, arXiv:2604.10508) across 7 models is that the first two rounds capture 76–95% of the achievable gain, with near-zero return from round three on; Self-Refine likewise plateaus by round 3. Extra rounds only multiply wall-clock and token spend. One further detail: the second round’s repair output is not re-verified inside the loop — the repaired draft is kept, and final verification is left to the user-side manual compile/export path.

A counter-intuitive design point: the repair is a full-document rewrite, not a scoped patch of the kind used in part 7. The reason is a property of LaTeX errors — they are often decoupled from the root cause, and line numbers are unreliable (LaTeX Compilation Challenges, arXiv:2603.02873), so pinpoint fixes at the reported line easily patch the wrong place. Scoped editing presupposes deterministic targeting, and compile errors are precisely where that presupposition fails.

Termination: fingerprint, round cap, wall clock

The biggest production risk of a self-repair loop is not failing to fix — it is failing to stop. The consensus on loop governance is that “when to stop” must not be left to the model’s discretion; an outer controller enforces it with deterministic criteria. There are three gates:

The no-progress gate. After each failed round, fingerprint the errors — the first 120 characters of the first ! error line (LaTeX’s anchor line); if it equals the previous round’s fingerprint, declare no progress, stop immediately, keep the current draft:

def _error_fingerprint(errs: str) -> str:
    """Compile-error fingerprint: first '! …' error line (LaTeX's anchor),
    else the first 120 chars. Used for no-progress detection."""
    for line in (errs or "").splitlines():
        s = line.strip()
        if s.startswith("!"):
            return s[:120]
    return (errs or "").strip()[:120]

If the same error survived a repair round, this model has no answer for this error; feeding it the same report again will most likely produce the same failure, and burning more tokens has no expected return.

The round cap. for attempt in range(2) — a hard limit.

The wall-clock gate. A single compile runs up to 4 subprocess passes at 180 seconds each — a token budget cannot contain CPU time. COMPILE_WALL_BUDGET (default 600 seconds) is checked before each round; on expiry, stop and keep the current draft. The config comment states the motive: prevent multi-round self-repair from hogging the customer machine’s CPU.

All three gates are deterministic criteria executed on the Python side; the model has no say in whether the loop stops — the same stance as the tool-loop circuit breaker in part 4.

Minimal repair context: ~8k tokens, not 40–60k

How the repair call’s context is constructed is the single most money-saving decision in this loop. The intuitive approach is to reuse the agent’s full exploration history — surely the model “fixes better knowing more”? The opposite holds. The judgment recorded in the code comment: 95% of compile errors are template-level syntax problems (balancing, command misuse, template variables), and fixing them requires exactly three things:

• the full current draft;
• the error excerpt (≤1800 characters);
• the formatting hard rules (no % comments inside \def braces; bold vectors with \symbf; no bold inside accents; balance braces/environments/$; end with a top-level \end{document}).

About 8k tokens in total. Reusing the full exploration messages would carry 40–60k tokens of dead weight — source material, tool-call records, multiple draft versions — none of it contributing to fixing one brace; switching to a fresh minimal context saves 30–50k tokens per repair round. The system prompt narrows the mandate to a single sentence:

You are a LaTeX compile-repair assistant: fix compile errors only — do not rewrite content, do not add or remove sections, do not touch data.

This is not only about cost: the measurements in Revisit Self-Debugging (arXiv:2501.12793) show label-only feedback beating verbose detailed feedback — inaccurate long context introduces ambiguity and lowers scores. Commercial products are isomorphic: Overleaf’s commercial Error Assist also deliberately minimizes the repair context, sending only the full error, the relevant code lines, and a file-name list (product-page information, not independently verified).

Concurrency isolation: a compile directory per session

A plain constraint you only learn by tripping on it: the compile working directory must be isolated per session. compile_tex writes article.tex into the directory, copies the cls and figures, and rmtree-rebuilds the figure subdirectory during assembly — two concurrent generation runs sharing one directory will overwrite each other’s source, delete each other’s figures, and pollute each other’s logs. After partitioning the repair directory as _repair/<session_id>, each run writes its own. The lesson generalizes: a verifier is a side-effecting process, not a pure function — when wiring an external verifier into a loop, isolation is a default, not an optimization.

The whole picture, quantified

Where not to build this loop

1. It presupposes a cheap, objective, unfakeable verifier. Compilers, type checkers, and unit tests qualify; “is the writing good” does not. The stop condition must anchor on an objective signal like “compilation succeeded” — never on the model’s self-assessment. Quality dimensions without a verifier belong to evaluation and judges (parts 8 and 9), not to this loop.
2. Deterministic repair comes first. Pitfalls that can be written as rules (known syntax traps, missing-figure placeholders, variable backfill) should be cured by the zero-token sanitizer before any LLM repair; the repair loop only backstops the long tail that rules cannot enumerate. Reversing the order means using a probabilistic component for deterministic work: paying a call every time while adding new uncertainty.
3. Weak models need a non-LLM exit. Survey data puts weak models’ repair rate on deadlocked errors at only about 30% (not independently verified). The loop must ship with a fallback that does not depend on the model — here, export_zip produces a compilable package to hand to Overleaf or a human. Without an exit, the loop degenerates on weak models into “burn the budget, then give up.”
4. When the verifier is expensive, the budget gate precedes the round cap. Here a single compile is wall-clock-bounded at 4 passes × 180 seconds (at most 720s), so K=2 is affordable; if the verifier is a ten-minute integration test suite, the same K=2 may be unacceptable — the wall-clock budget must become the first gate, including the option of “give up without repairing at all.”

References

• Self-Refine: Iterative Refinement with Self-Feedback (arXiv:2303.17651) — self-reflection is nearly useless on tasks where errors are hard to self-detect; gains return with external signals; benefit concentrates in the first 1–2 rounds (its “+20% average” figure not independently verified).
• When Can LLMs Actually Correct Their Own Mistakes (TACL 2024) — prompting-only self-critique does not improve, and can degrade, arithmetic/code tasks; flattering self-correction results mostly relied on an oracle.
• How Many Tries (arXiv:2604.10508) — the first two rounds capture 76–95% of achievable gain; near zero from round 3.
• LaTeX Compilation Challenges (arXiv:2603.02873) — LaTeX errors are often decoupled from root causes; line numbers are unreliable.
• Revisit Self-Debugging (arXiv:2501.12793) — label-only feedback beats verbose feedback.
• Overleaf Error Assist — the commercial isomorph of minimal repair context (product-page information, not independently verified).