27 days of AI-assisted development: a retrospective in numbers

Also in 中文. Part 14 of the Engineering LLM Applications series — the closing installment. The previous thirteen articles discussed mechanisms; this one returns to the 27 days that produced them.

Scope and accounting rules

From 2026-06-10 to 07-06, one person plus a coding agent (Claude Code) built, from scratch, the AI writing agent this series has been citing throughout. Because the project enforced a paper-trail rule from day one — every development conversation produces a devlog entry — this retrospective does not rely on memory. The data comes from 94 devlogs, git history, a count of the test directory, and a line-by-line sum of the research experiment’s usage.jsonl. The article is organized as four ledgers: output, discipline, debt, and conclusions. Approximate figures state their accounting basis; anything the logs marked as “suspected cause” stays hedged here.

The output ledger

Two notes. First, output came in bursts: of the 94 devlogs, 27 landed on 06-15 alone and 14 on 06-11 — at the granularity of “one conversation advances one feature”, parallel sprints became the norm. Second, the third pivot (SaaS → standalone) is worth recording as method: every change was flag-gated and off by default; on landing day, all 47 backend tests passed in the default configuration — zero regression in SaaS behavior. The test baseline then grew to 344 passed (devlog figures), with the repository ultimately counting 351 test functions.

The discipline ledger: which conventions paid for themselves

Of the working conventions written into CLAUDE.md on day one, four were repeatedly proven worth their cost by day 27.

A devlog for every conversation. Fixed format: goal / what was done / key decisions / files changed / leftovers. The direct payoff is this article — a retrospective that needs no recollection. The larger payoff is attribution: the venv-pollution debt described below can be reconstructed as a complete causal chain from two log entries one hour apart. The research experiment ran the same regime — journal entries are append-only and machine-readable results go into runs.tsv — which is what allowed the summary document to state that “all numbers come from recorded experiments.”

Git as the safety net: commit every state that runs. The very first conversation ended with git init and a commit (f705076, 115 files). None of the three pivots ever lost work: the standalone pivot branched from the current HEAD with zero disturbance to the working tree, preserving the experiment branch’s 20 commits intact. “If the AI breaks it, roll back” went from slogan to daily practice over the 27 days.

base_docs is read-only. Domain material, few-shot samples, and test data live in a read-only zone; generated artifacts go into separate directories. Through three pivots and dozens of generation rounds, the evidence base was never contaminated.

One feature per change. Twenty-seven devlogs in one day is not 27× productivity; it is 27 scope-narrowed conversations, paired with “one commit plus tests per feature.” This is the deterministic boundary applied isomorphically to the development process itself: narrow the scope of every change, and defend existing assets with deterministic gates — tests and git.

The debt ledger: verification debt and environment debt

Verification debt, case one: a batch verification that failed wholesale. On 07-05, to benchmark the agent pipeline against the state of the art, a 105-agent web-research workflow was run. Its adversarial verification stage failed as a whole batch when the subscription’s session quota was exhausted: all 25 claims came back unverified. The only remedy was manual: pick the 3 papers carrying the most weight — and most likely to have been fabricated by a retrieval agent — and check them line by line against the arXiv originals. One had its core numbers confirmed (revisions regress 16–27% of already-covered content); one was genuine but required two corrections to how it had been cited; one was architecturally real, with its specific numbers absent from the abstract and therefore left marked unverified. The machine produced 25 conclusions in one evening; human verification covered 3. That gap is the shape of verification debt.

Verification debt, case two: a conclusion frozen for want of a human anchor. The 4-day controlled experiment obtained highly direction-consistent results under LLM-judge scoring (see part 12); the 30-pair anonymized blind-evaluation sheet was exported, and the machine’s verdict on the same items was B=20 / A=3 / tie=7. But human scores were never collected. The summary document therefore had to settle for: H1 is “a strong candidate with method-level evidentiary support — not established.” The one step that 3,511 calls and ~7.94M tokens could not buy was precisely the only step independent of the machine.

Environment debt: one experiment’s contamination, detonating an hour later. On 06-23, minted code highlighting was trialed for the standalone build; the conclusion was that it is unworkable under frozen distribution, so the build reverted to the safe default. But the experiment’s pip installs had touched the shared venv. One hour later (by log timestamps), the packaged desktop app failed to start; one layer of the four-layer root-cause analysis was a missing pydantic_core — the log annotates it as “suspected cause: the minted detour’s pip installs overrode the pydantic version.” The recorded lesson, verbatim: clean up the venv after temporary experiments.

The methodological conclusion: the bottleneck moved to verification and judgment

The output ledger shows AI assistance expanding the engineering scale one person can drive by an order of magnitude — a full-stack application, desktop distribution, 351 tests, and a controlled experiment advancing simultaneously within 27 days. That is a case judgment against the author’s own manual pace, not a controlled measurement. But the three debts point to a single conclusion: once scale expands, the bottleneck moves from production to verification and judgment. Twenty-five claims needed a human reading the originals; H1 needed 30 human blind evaluations; the venv contamination needed a human-driven layer-by-layer diagnosis. Every conclusion the machine produces needs an anchor independent of the machine — deterministic tests, git history, arXiv originals, human scores. The mechanisms discussed across the first thirteen articles of this series are, in the end, ways of manufacturing such anchors at scale.

Where this retrospective does not apply

1. N=1. One developer, a greenfield project, no collaboration friction, no legacy code. These numbers are a record, not a statistic; extrapolating to teams or brownfield systems has no basis here.
2. Output metrics are not outcome metrics. Lines, tests, and devlogs measure production volume; the 27 days include no user data, so product value went unmeasured.
3. Devlog count is not workload. Devlogs count conversations; a 27-entry day reflects short, scope-narrowed sessions, not 27× throughput.
4. The token figures cover only the experiment’s metering. usage.jsonl records the research experiment’s API calls only; the coding agent’s own usage during main-app development is not metered there, so total cost cannot be extrapolated from these numbers.
5. “Zero regression” was purchased in advance. Three pivots without incident rested on flag-gating plus a test baseline; without that baseline, the same speed produces silent breakage.

References

• The three papers manually re-verified after the 105-agent workflow: Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (arXiv:2601.13217; core numbers checked against the original); PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing (arXiv:2604.05018; genuine, with two citation corrections); RP-ReAct (arXiv:2512.03560; architecture genuine, specific numbers not independently verified).
• Experiment data and statistical basis: the writing-judgment-distill SUBMISSION_SUMMARY and results/usage.jsonl.