The deterministic boundary: a first principle for LLM application architecture

LLMs are probabilistic components; production applications need engineering guarantees. This article states a field-tested architectural principle — never ask the model what code can answer, let the model produce only minimal fragments, and gate every fragment deterministically — with five mechanisms, measured gains, and the limits of its applicability. The evidence base is a production-grade AI writing agent with 351 tests.

Also in 中文. Part 11 of the Engineering LLM Applications series — where the mechanisms of the preceding ten articles converge into one principle. The series is grounded in the implementation and measurements of a production-grade AI writing agent (FastAPI + React + Tauri, 351 test cases).

The problem: probabilistic components, engineering obligations

Once an LLM is embedded in a production application, the hard engineering problem is not that the model is insufficiently smart. It is that a probabilistic component degrades system properties in several predictable ways:

Scope escape. Ask the model to “fix one section” and it rewrites the whole document. Empirical measurement of full-rewrite revision (arXiv:2601.13217) shows revisions regress 16–27% of already-covered content on average — not a model slip, but a statistical property of open-ended rewriting.
Loop collapse. In long contexts, agents degenerate into calling the same tool with identical arguments. A public case (qwen-code issue #4695) records 43 consecutive git status calls consuming 8.9M tokens in one session; SDK-level retries cannot help, because every call succeeds at the API layer.
Cost drift. If every turn carries the full context, a “fix the keywords” request pays for the entire document plus all source material — we measured roughly 6–7k input tokens for such requests, of which about 1.5k carries information.
Untestable regressions. Non-deterministic output defeats conventional assertions; a prompt edit silently shifts quality and CI has nothing to catch it with.

Against these failure modes, our project converged on one architectural principle, which this article calls the deterministic boundary:

Never ask the model a question deterministic code can answer; let the model produce only the minimal necessary fragment; admit no fragment into system state without passing a deterministic gate.

The principle is orthogonal to two existing families of tooling: structured outputs / function calling address format determinism (what the output looks like); guardrail frameworks address content compliance (what the output may say). The deterministic boundary addresses scope and state determinism — which part of the system the output is allowed to modify, and under what conditions it lands. The str_replace / fast-apply pattern in code editors (model produces the diff, application is deterministic) is the same principle expressed in another domain.

The rest of this article walks through five mechanisms, using document revision as the running example. The stack is Python/FastAPI and the document format is LaTeX, but the mechanisms are format-agnostic: they apply to any domain with parseable structure.

Mechanism 1: deterministic targeting — the sectionizer

Scoped editing presupposes a deterministic answer to “what is the character range of the target section.” Our sectionizer (~260 lines, zero LLM calls, zero dependencies) parses LaTeX source into a node tree along \section / \subsection / \subsubsection, assigning each node a char_start / char_end. Two design points deserve expansion:

Mask, don’t shift. A \section inside a comment or a verbatim environment is a false heading — but deleting those regions before parsing would invalidate every character offset. The solution is to replace them with equal-length whitespace:

def _mask(content: str) -> str:
    """Replace comments and verbatim environments with equal-length spaces
    (all character offsets preserved), so a \section inside them is not
    treated as a real heading — while splice still uses original offsets."""

Headings are located on the masked text; content is read from the original at the same offsets. Parsing and splicing share one coordinate system, so there is no translation step — and no translation bugs.

Dual anchors. Every node carries both a positional anchor (n0, n1, stable within one version) and a content-fingerprint key (L1:ErrorAnalysis, derived from level + title). Positional anchors drift when sections are inserted or removed; content keys follow the title and survive across versions, so they take precedence during resolution. This is the cheapest possible mechanism for “refer to the same section across versions”: no diff algorithm, just two string keys.

The failure mode is deterministic too: if no structure parses, the sectionizer returns editable=False and the caller falls back to whole-document revision. The boundary machinery never raises into the main flow — its job is to narrow the model’s scope, not to add a failure point.

Mechanism 2: scoped generation and deterministic splicing

After targeting, the prompt sent to the model contains only the target section (plus hard rules against scope escape), and the model returns only a replacement fragment for that section. The fragment’s path back into the master document is entirely code:

strip_fragment — cleanup: remove Markdown fences; if the model disobeyed and returned a full document, cut out the span from the first heading to just before \end{document};
check_fragment — health check: balanced braces, balanced environments, no \documentclass (soft signals that warn rather than block);
splice(content, start, end, new_text) — a one-line reassembly in which everything outside the range is byte-identical.

Byte-identical is a testable promise, not an adverb: the assertion is written directly into the test suite. One implementation lesson: a fragment must preserve the count of top-level \section commands within the replaced span, otherwise the positional anchors of subsequent edits in the same round drift and hit the wrong section. That bug was caught by an internal adversarial review and is now pinned by a permanent test.

Mechanism 3: zero-token pre-routing

“Which part does this feedback want to change” is a routing problem. The default is to ask a model, but a substantial share of feedback carries an unambiguous explicit target: “the bibliography format is wrong”, “the abstract is too long”, “the units in Table 2 are off”, “rewrite §3.1”. Those are decided by rules:

signal classes = { bibliography terms, abstract/keywords, section references
                   (Table N / Figure N / § numbers / unique section titles) }
exactly one class hit  → route directly (0 tokens)
no signal / conflicting classes / "whole document" wording → fall back to LLM routing

The design stance is conservative-first: prefer falling back to the LLM over ever guessing. Disambiguation is deterministic as well — when both “Data Processing” and “Data Processing and Results” match, the longer title wins. Combined with the narrow prompt from Mechanism 2, this cut input tokens for keyword-fix-type requests from roughly 6–7k to about 1.5k in our measurements; routing itself costs nothing, adds no latency, needs no API key, and runs at full speed in offline tests.

Mechanism 4: the zero-token regression gate

The autonomous agent’s reflection loop (draft → self-revise × N) is where scope escape does the most damage. Beyond constraining revision into a structured edit plan (schema-enforced changes[], where an empty array means converged and terminates early) plus scoped splicing, there is one final gate before anything lands — regression_check, six fully deterministic criteria:

loss of the compilable skeleton (\documentclass / \end{document});
length collapse below 60% of the previous version (truncation, or “rest unchanged”-style elision);
structural score decrease (eight equal-weight checks: section count, error analysis, data tables, equations, no TODO placeholders, …);
shrinkage of the \cite key set;
fewer sections;
fewer figures or tables.

Any trigger reverts to the previous version and stops the iteration. The reason for choosing deterministic criteria over one more LLM judge call is stated in the module’s own comments: this particular failure mode — losing content — is deterministically detectable, whereas the pairwise judge’s validity in this domain is unverified and would cost an extra call per round. More correct, and cheaper.

The gate has its own edge case: bibliography generation runs after reflection, so with an empty bib the “citations closed” check is false for any draft containing \cite, and the gate would misread “added a citation” as degradation. The fix excludes that check from the gate score while the bib is empty; citation loss is still caught separately by the key-set-shrinkage criterion. A gate is code, and code has bugs — but a gate’s bugs can be pinned by unit tests, while model drift cannot. That asymmetry is precisely why the gate belongs on the deterministic side.

Mechanism 5: putting the boundary itself under CI — prompt-shape gates

Offline tests cannot run a real LLM, so they cannot measure generation quality — but they can test the shape of a prompt: whether the scoped prompt still demands “change only this section”; whether the assembly function’s output on gap-free input is byte-identical to a golden file. If someone edits the prompt back to “output the full revised document”, plain pytest goes red. The most fragile part of the deterministic boundary — constraints written in natural language — thereby gets the same regression protection as code. A companion gate validates the scorer itself: gold-standard good reports must score ≥ 0.75, bad ones ≤ 0.35, with a gap ≥ 0.4 — the scorer must demonstrably encode quality before it is allowed to act as a gate.

The whole picture, and the measured gains

flowchart TD A[Edit request] --> B{Deterministic pre-router} B -->|explicit target hit| S[Scoped prompt: target section only] B -->|no signal / conflict| L[LLM router] --> S S --> G[LLM generates section fragment] G --> C[Deterministic cleanup and health check] C --> P[Splice: bytes outside range unchanged] P --> H{Who is in the loop} H -->|conversational editing| U[Proposal ledger: lands only on user accept] H -->|autonomous reflection| R{Zero-token regression gate} R -->|pass| V[Commit as new version] R -->|degraded| X[Revert to previous version, stop]

Mechanism	Alternative	Measured gain
Sectionizer + splice	Full-document rewrite by the model	Bytes outside the target range unchanged; avoids the 16–27% content-regression mode
Zero-token pre-router	LLM routing on every request	0-token decisions for explicit targets; input 6–7k → ~1.5k tokens
Regression gate	Accept every revision / LLM judge	One judge call saved per round; degradation reverts, convergence stops early
Prompt-shape gate	Manual review of prompt edits	The deterministic share of 351 tests runs fully offline, in CI

Where the principle does not apply

In keeping with this series’ convention, the honest boundary of the boundary:

It presupposes parseable structure. LaTeX has \section, code has ASTs, Markdown has headings; free-form prose has none. Without structural anchors there is no sectionizer — the boundary collapses to just cleanup and gates.
The boundary is code, and code has carrying costs. The sectionizer must handle starred headings, optional short-title arguments, comment and verbatim masking; we maintain dozens of unit tests for this layer alone. If the product’s shape is still changing fast, building the boundary early makes every change cost twice.
It protects stock, not flow. The boundary’s value grows with the value of existing assets: it prevents version N from being wrecked, and contributes nothing to generating version 1. Tasks that are creation-dominated, with no existing asset to protect, do not need it.
Deterministic criteria only measure the measurable. The regression gate prevents getting worse; it cannot certify getting better. Preference-level quality still requires pairwise judges or humans — which is the subject of another article in this series (validity engineering for LLM-as-judge).

References

Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (arXiv:2601.13217) — empirical measurement of the 16–27% content-regression mode in full-rewrite revision; structured edit plans and scoped revision are the remedies it names.
PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing (arXiv:2604.05018) — structured multi-agent pipelines beating autonomous baselines (+50–68% / +14–38%); its accept-or-revert refinement gate reports 79–81% win rate with 0% degradation (body-text detail, not independently verified).
Self-Refine: Iterative Refinement with Self-Feedback (arXiv:2303.17651) — feedback must be specific and actionable; generic feedback measurably hurts, and tasks where errors are hard to self-detect need external signals.
qwen-code issue #4695 — a public case of tool-loop collapse (43 identical calls / 8.9M tokens).