Multi-agent adversarial review as an engineering practice

Also in 中文. Part 13 of the Engineering LLM Applications series.

The problem: the typical failure of an LLM finding is that it looks real

When you use an LLM to review code, volume is never the bottleneck — it will happily list fifty “potential issues” against any codebase. The bottleneck is at the other end: the typical failure mode of an LLM finding is plausible-but-wrong. The line number is specific, the failure narrative is complete, the suggested fix is fully formed — and then it falls apart the moment you confront it with the code: the guard already exists, the alleged race path is unreachable, the “leaked” resource is reclaimed somewhere else.

A review pipeline without a verification stage drowns in fake findings, and fake findings cost you both ways: either you falsify each one by hand, paying back every hour the review saved, or you start ignoring the report wholesale — and the real problems die with the fake ones. The second outcome is the more common and the more fatal: a report that mixes true and false findings carries close to zero engineering information.

A separate, independent problem is breadth: a single-perspective review — whether a human or a single agent — cannot stay simultaneously alert across security, concurrency, billing, and cross-platform consistency.

Both problems point at the same structural remedy: split “find problems” and “verify problems” into two stages with opposite objectives — the finding side optimizes for coverage, the verifying side optimizes for truth. This article walks through the structure, grounded in two production runs (one full-project audit, one session-level review) plus one set of overturn data.

The structure: parallel discovery by dimension, adversarial verification per finding

Stage one: N auditors in parallel, each guarding exactly one dimension. The full-project audit used 6 (backend core, API-DB-security, LaTeX-verify-distill, frontend, test blind spots, documentation drift); the session-level review used 4 (concurrency/runs, security, frontend regression, data integrity). The point of slicing by dimension is not parallel speed-up but forced perspective: an auditor that only looks at concurrency does not have its attention taxed by functional correctness.

Findings obey a format discipline: every one must land as file:line + a concrete failure scenario + a proposed fix. This is the precondition for everything downstream — it turns a finding from “something feels off” into a falsifiable claim.

Stage two: each finding goes to an independent verifier. The verifier is not the agent that produced the finding and does not enter with that agent’s reasoning in context; its instructions are default skepticism, specialize in falsification — go into the code and look for evidence that the finding does not hold. The verdict is three-valued: CONFIRMED / uncertain / dismissed, and only CONFIRMED enters the repair queue.

Independence is a requirement, not an option. When one agent checks its own work, its finding is sunk cost sitting in its context, and it will defend it; only when “successfully overturning the finding” is the verifier’s success criterion do the incentives align. This is the same issue as judge validity from part 9 of this series, seen from the other side: the verifier is itself a judge, and its validity has to be demonstrated by the fact that it actually overturns things — data below.

Two measured runs

Beyond the counts, what deserves attention is the type of problem on the confirmed list — a substantial share is the kind a single person reading the diff end-to-end would very likely miss:

• TOCTOU race in the 409 concurrency guard (session-level): “check whether a run is already running” and “create the run” were not atomic, so concurrent requests could double-start. The fix moved the check, the creation, and the handle registration together inside runs._LOCK; the repair was closed out empirically — two concurrent requests return [200, 409].
• Chat billing used the unresolved provider (session-level): with DEFAULT_PROVIDER=anthropic and the request omitting the provider field, cost was computed at deepseek’s unit prices — roughly a 60× underestimate. Provider resolution lives in one place, pricing in another; reviewed module by module, both sides “look correct”.
• Anthropic streaming max_tokens=32000 while DeepSeek’s was 64000 (full audit, medium): long reports silently truncated on one provider; only a cross-provider consistency comparison exposes it. Fixed by unifying on _STREAM_MAX_TOKENS=64000.
• Figure sandbox environment hardcoded POSIX (full audit, high): on the Windows distribution (the primary platform), the CPython subprocess cannot start for lack of SystemRoot/TEMP, so make_figure crashes unconditionally.
• EXPOSE_RESET_TOKEN defaulting to true (full audit, high): any anonymous caller could obtain a reset token and change a password — account takeover; changed to a secure default of false.

These respectively require concurrency-interleaving reasoning, cross-module dataflow tracing, cross-provider comparison, and cross-platform runtime knowledge — exactly the categories that one dedicated perspective per dimension covers and one sequential read-through misses.

One disposition detail worth copying: the 3 deferrals are not silent omissions but explicit decisions with recorded reasons — all are SaaS-only issues neutralized in the shipped STANDALONE configuration by _no_cloud/forced-local, and fixing them outright would have meant disturbing a set of non-STANDALONE test semantics; they are filed for whenever SaaS is revived. The output of a review is not just fixes; it is also this traceable record of why not.

Verification is not a rubber stamp: overturn data

Whether the adversarial verification stage actually works needs evidence of its own. The same find-then-adversarially-recheck structure was applied to a different kind of object: auditing whether the writing rules produced by the distillation experiment (part 12 of this series) genuinely learned “judgment” rather than reskinning structural templates. Eleven agents in total: for each rule pair, an item-by-item initial classification, followed by an adversarial re-review whose instructions were, again, default skepticism — specializing in items whose “because” is a tautology or whose structural action carries only a thin veneer of justification.

The initial numbers of all five pairs were revised downward; the post-adversarial mean is 0.746, and the overturned items follow one highly consistent pattern — the action overlaps a structural template and the stated reason is close to a tautology. The pair-level verdicts held 5/5, but no pair emerged untouched. The original report also flags, unprompted, that pair 5’s 0.9 is optimistic (the initial 1.0 was itself suspicious).

This data establishes two things. First, the verifier really does overturn items case by case — it is not process decoration. Second, the value of the adversarial stance runs in both directions: in code review it kills false positives (2 dismissed; “all others falsified” at the session level), and in evaluation tasks it deflates optimistic self-assessment (100% → 90%).

Cost and applicability

The cost structure is straightforward: every finding pays for one additional dedicated verification pass, plus orchestration overhead. When it is worth it:

• Large change surfaces. The session-level run covered 44 files / 1,467 lines including a core-path refactor (“merging chat into run”) — a surface beyond what one person can hold in a single pass.
• Pre-release / security surfaces. Problems like EXPOSE_RESET_TOKEN or the Stripe webhook skipping validation when no secret is configured (fixed to fail closed: 503 without a secret) have a miss cost wildly out of proportion to the verification cost.
• Legacy code never systematically audited. The full-project audit’s stated goal was precisely “not just the new code — focus on the broader/older parts”, code outside the field of view of any recent review.

When it is not worth it, the source record already contains the judgment: single-feature implementation is “sequential and interleaved, with no fan-out” — solo work suffices; review and audit (breadth plus independent verification) are what the multi-agent workflow is actually good at. Extrapolating to small changes: on a few dozen lines of diff, falsifying three findings by hand is far faster than orchestrating a six-auditor pipeline, and the verification overhead simply exceeds the benefit.

Limits

1. The shared-origin blind spot. The verifier and the author of the code under audit are the same (or a sibling) LLM. The adversarial structure removes positional bias — no one defends their own output anymore — but it does not extend the capability boundary: a category of problem invisible to both sides stays invisible to the verification stage too. It is no substitute for signals of a different origin — actually running the code, live concurrency tests, human domain knowledge.
2. “0 uncertain” does not mean “0 escaped”. 27 CONFIRMED with 0 uncertain measures the precision side: nearly everything raised was real. It says nothing about the recall side — the problems no auditor raised at all. This very audit confirmed three test blind spots: an audit can find the holes in the tests, but there is no instrument measuring the holes in the audit.
3. Verification depth is itself sampled. The overturn experiment’s report states honestly that only pair 4’s source file was empirically spot-checked; the rest relies on sub-agent labeling. Verification can be shallow, and a CONFIRMED stamped by shallow verification deserves a discount — mistaking “verification happened” for “verification was thorough” is this pipeline’s most seductive self-deception.
4. The structure depends on finding discipline. file:line + failure scenario + fix is what makes a finding a falsifiable claim; relax it to “module X may have issues” and the verifier has nothing to confront — the whole pipeline degenerates into two rounds of opinion exchange.