Controlled revision in reflection loops

Also in 中文. Part 10 of the Engineering LLM Applications series.

The problem: why “draft → self-revise × N” often makes things worse

The standard closing act of an autonomous generation pipeline is a reflection loop: once the first draft exists, let the model critique and revise it for a few rounds. Intuition says more rounds can’t hurt. Our first implementation was a direct transcription of that intuition — critique produced a prose commentary, revise rewrote the whole document accordingly, and the new draft replaced the previous one unconditionally:

msgs.append({"role": "assistant", "content": draft})
yield {"type": "draft", "text": draft}      # keep this version unconditionally

Empirical measurement says the intuition is wrong. arXiv:2601.13217 (Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision — we read the arXiv original to verify it) measured multi-turn revision in deep research agents: when handling feedback, full-rewrite revision regresses 16–27% of already-covered content on average, with citation quality degrading alongside. More alarming: unguided “please reflect and revise yourself” is ineffective for almost every agent tested — only 1 of 5 improved (+3.6%) — and it causes more collateral damage than targeted feedback. On the mechanism side, Self-Refine (arXiv:2303.17651, retrieval excerpt, not adversarially verified) is consistent: feedback must be specific and actionable, generic feedback measurably hurts, gains concentrate in the first 1–2 rounds, and tasks where errors are hard to self-detect see almost no benefit. The positive contrast is PaperOrchestra (arXiv:2604.05018): its refinement loop carries an explicit “accept only if better, revert and stop if worse” gate and reports a 79–81% win rate with 0% degradation (a body-text detail, not independently verified).

Held against that evidence, our first implementation stepped on all three mines:

The three sections below walk through the rework in order of increasing invasiveness. The context is a LaTeX report-generation agent (scan → explore → figures → write → reflect × N → bib → compile self-repair), but the mechanisms are format-agnostic.

Rework 1: reflection outputs a structured edit plan

The first cut lands on the shape of the feedback. The critique is no longer prose; it is forced into a JSON edit plan (prompt translated here — the production prompt is Chinese):

_REFLECT_SYS = (
    "You are a reviewer of lab reports / course notes. Check the current draft "
    "against the source material and identify only problems that genuinely need "
    "fixing, locating each one to a specific section (give its anchor, e.g. n2); "
    "use \"whole\" for cross-section / overall-structure / abstract-and-keywords "
    "issues. Each item carries an issue and an actionable action. Do not mention "
    "what is already fine; no generic polishing. Output strictly JSON: "
    "{\"changes\":[{\"section\":\"n2 or whole\",\"issue\":\"\",\"action\":\"\"}]}; "
    "if the draft is acceptable overall, output {\"changes\":[]} and nothing else."
)

Four design points:

1. Every issue must be located to a specific section (the anchor assigned by the sectionizer, e.g. n2); cross-section, overall-structure, and abstract/keywords issues are explicitly marked "whole". Location is not decoration — it decides which application path the item takes next.
2. Issue and action come as a pair. “The error analysis is shallow” alone is not enough; an executable action is required — the direct application of Self-Refine’s “feedback must be specific and actionable”.
3. The empty array is allowed. {"changes":[]} means “acceptable overall” and terminates the reflection loop early. Revision flips from “change by default” to “no change by default, change only with cause”.
4. Parse failure never breaks the flow. _parse_reflect_plan returns an empty list for non-JSON or missing changes, which the caller treats as “no changes needed”; plans are capped at 8 items; the title fallback when an anchor misses is “exact match first, then the longest contained title” — so “Results” cannot steal a match meant for “Results and Discussion”.

A companion change: reflection now runs on a fresh minimal context (system + source material + current draft) instead of dragging along the full exploration transcript; the material sits as a stable prefix and hits the cache across rounds.

Rework 2: pure section-level issues go through scoped per-section splicing

Each plan item is routed by its location. Pure section-level issues no longer trigger a full rewrite; they reuse the mature machinery of the interactive editing side (sectionizer, scoped prompts, fragment health check, deterministic splice — the toolkit of Part 07), applied section by section. For each item, _scoped_reflect_apply re-parses and re-resolves the target on the current working text (offsets have moved after the previous splice), sends the model only the target section plus that item’s issue/action plus relevant material, and health-checks the returned fragment (non-empty, balanced braces, balanced environments, no \documentclass). A fragment that fails the check causes that section to be skipped — a bad fragment is never spliced back. A fragment that passes is spliced in, with every other section byte-identical. Only cross-section issues, abstract/keywords issues, or drafts that cannot be sectionized fall back to a whole-document minimal rewrite.

This step has one implementation pitfall, found by an internal adversarial audit (rated [medium]):

# The fragment must preserve the count of top-level \section commands within the
# replaced span (usually =1; =0 for a subsection) — otherwise the splice changes
# the document's section count, the positional anchors of subsequent scoped items
# in the same round drift and edit the wrong section, and the "added sections"
# direction is not caught by regression_check (it only reports decreases).
# Count mismatch → skip this section; never splice a fragment that would shift things.
if len(re.findall(r"\\section\{", new_text)) != len(re.findall(r"\\section\{", section_src)):
    continue

The model occasionally gets “helpful” and splits one section into two: the fragment itself is healthy and passes every check, but after splicing, the document’s section count goes up by one, the positional anchors of all subsequent plan items in the same round drift, and the wrong sections get edited — while the regression gate only reports section decreases, leaving this path with no backstop. The conservation check closes it off before the splice.

Rework 3: the zero-token regression gate

Whether a round went through scoped splicing or the whole-document fallback, its output passes regression_check before landing — six criteria, all deterministic, zero API calls:

• loss of the compilable skeleton (\documentclass / \end{document});
• length collapse below 60% of the previous version (truncation, or “rest unchanged”-style elision);
• structural score decrease;
• shrinkage of the \cite key set (with the lost keys reported);
• fewer sections;
• fewer figures or data tables.

Any trigger means regressed: revert to the previous version and stop reflecting. If the two versions are character-identical after whitespace stripping, the result is converged — also an early stop, so convergence wastes no rounds.

Why not use an LLM judge as the gate? The reason is written in the module’s own comments: this particular failure mode — losing content — is deterministically detectable, whereas the pairwise judge’s reliability in the Chinese domain is unverified (the judge module’s own caveat) and would cost one extra real call per round. More correct, and cheaper. The judge’s proper place is offline evaluation, not an online gate; the validity question is the subject of Part 09.

The gate has its own edge case: in the pipeline, bibliography generation runs after reflection, so during reflection the bib is always empty — which makes the “citations closed” check false for any draft containing \cite, and a newly added citation would be misread as a structural-score drop, wrongly reverting a good edit. The fix: while the bib is empty, that check is excluded from the gate score, and citation loss is judged separately by “the key set must not shrink” — pinned by a dedicated regression test (empty bib produces no false positive).

Gate decisions are surfaced as persisted step events (visible in the log page and across reconnects), with a structured reflect_gate field attached to the trace so the experiment bench can observe whether reflection is actually improving quality and which round reverted.

The reworked loop, end to end

Tests arrived in batches with the rework: when the gate landed, test_eval_report +7 (accept / revert / converge / lost section / lost citation / length collapse / empty-bib no false positive) and test_run_agent_mock +1 (degraded revision → revert to previous version and stop), with 303 backend tests green; 306 after scoped reflection landed; 322 by the end of this round of work (plus 19 on the experiment bench). Generation quality itself cannot be tested offline — but every deterministic component of the loop can: plan parsing, routing, splicing, the gate.

Lessons from the research process itself

The evidence chain behind this rework deserves its own post-mortem. To benchmark against the state of the art, we ran a 105-agent networked research workflow; the adversarial-verification stage failed wholesale when the subscription session quota was exhausted, leaving all 25 claims stuck at unverified. So we manually re-verified the 3 most load-bearing papers — the ones a retrieval agent would most plausibly fabricate — by reading the arXiv originals. 2601.13217: real, core numbers confirmed. PaperOrchestra: real, but with two corrections — it is not a Google paper, and its abstract describes itself as flexible rather than rigidly fixed, so it can only support “structured multi-agent division of labor beats autonomous loops” (win rate +50–68% / overall quality +14–38%), not “the more rigid the better”. RP-ReAct (arXiv:2512.03560): the architecture is real, but the specific scores in the retrieval claim about “being beaten by vanilla ReAct on simple tasks” do not appear in the abstract — plausibly extracted or confabulated from the body text by the retrieval agent, so they are marked unverified. One more calibration correction: the retrieval claim said “break rate 21–31%”, which is the body-text framing; the abstract says 16–27%, and this article uses the abstract’s figure.

The lesson is isomorphic to the article’s thesis: a retrieval agent’s output needs an accept/revert gate of its own — manual verification is the regression_check of the research workflow.

When not to do this

1. The gate’s criteria are proxies: they prevent getting worse, they do not certify getting better. Section counts, citation sets, and length all measure structural preservation; content that is wrong but structurally intact passes the gate. Preference-level quality still requires offline evaluation and judges — this gate does not replace them.
2. Scoped reflection presupposes sectionizable structure. A draft that cannot be parsed goes straight down the whole-document fallback — that fallback is not a defect, it is an acknowledgment of the boundary.
3. When the structure itself is wrong, scoped editing is powerless. Problems requiring global reorganization (moving sections, merging sections, changing the narrative line) must be marked whole and take the full rewrite; forcing them through per-section edits produces drafts that are locally fluent and globally incoherent.
4. Do not bolt this gate onto loops that already have an external signal. The compile self-repair loop has the compiler as its referee and stops on no-progress error fingerprints — a different kind of closed loop, and already the right design; adding a structural gate to it is redundant construction.
5. Keep N small. Self-Refine reports gains concentrated in the first 1–2 rounds (not adversarially verified), and 2601.13217 shows the break rate is the bottleneck of multi-turn revision — budget spent on a fifth reflection round is most likely paying for a revert.

This article pairs with Part 11, “The deterministic boundary”: that article promotes the gate and the scoped machinery into a unified architectural principle; this one is the complete rework record of that principle applied to a single closed loop — the reflection loop.

References

• Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (arXiv:2601.13217) — full-rewrite revision regresses 16–27% of covered content on average with citation degradation; unguided self-reflection is nearly useless and causes more collateral damage; its named remedies: structured edit plans and a dedicated Reviser. Verified against the original.
• PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing (arXiv:2604.05018) — structured multi-agent pipelines beat autonomous baselines by +50–68% / +14–38% (verified); its accept-or-revert refinement gate reports a 79–81% win rate with 0% degradation (body-text detail, not independently verified).
• Self-Refine: Iterative Refinement with Self-Feedback (arXiv:2303.17651) — feedback must be specific and actionable, generic feedback measurably hurts; gains concentrate in the first 1–2 rounds; tasks where errors are hard to self-detect need external signals. Retrieval excerpt, not adversarially verified.
• Reason-Plan-ReAct (arXiv:2512.03560) — plan/execute decoupled architecture (verified); the specific scores for “beaten by vanilla ReAct on simple tasks” are absent from the abstract and unverified.