The token economics of LLM applications

Also in 中文. Part 1 of the Engineering LLM Applications series. The series is grounded in the implementation and measurements of a production-grade AI writing agent (FastAPI + React + Tauri), and discusses transferable engineering methods.

The problem: why does fixing one keyword cost 6–7k tokens

When you build an LLM application, the first constraint that forces an architectural decision is usually not model capability — it is the bill. Our project (a LaTeX lab-report writing agent) received a blunt question from a user: “why did you send 6k?” — he only wanted to change the report’s keywords.

The precise ledger (before the fix):

The root cause is not the model but one-size-fits-all context assembly: whatever the edit, the full set of quality anchors plus the full material gets stuffed in. Fixing a keyword has no use for a complete exemplar report about a different experiment, and even less for raw data matrices. The user’s follow-up became the design charter for this pipeline: “the process may be complex, but the goal is to save tokens.” Cost optimization is not cutting features; it is layering context by problem type — minimal context per type. Below are the four layers we shipped.

Layer 1: zero-token deterministic pre-routing

“Which part does this feedback want to change” is a routing problem, and the default is to ask a model. But a substantial share of feedback carries an unambiguous explicit target: bibliography, abstract, keywords, “Table 2”, “Figure 3”, “§3.1”, a uniquely matching section title. These are decided with regexes and string matching (_fast_route — zero LLM calls, no API key even needed): exactly one signal class hit → route directly; “whole document” wording, no signal, or conflicting classes → fall back conservatively to LLM routing, never guess.

The gain is more than one saved routing call: the decision has zero latency, is 100% explainable (the log is tagged “deterministic fast decision, 0 tokens”), and offline tests run at full speed. This pattern is developed further as part of the deterministic-boundary principle in Part 11.

Layer 2: tiered system prompts

The most expensive part of the system prompt is the quality anchors: the writing methodology plus the golden-sample exemplar, about 3.2k characters, roughly 2k tokens. They are indispensable for whole-document generation — an early refactor once dropped them and generation depth and style collapsed immediately — but useless for small edits like “fix one section” or “fix the keywords”.

So build_system(scoped=True) adds a tier: paragraph-level and metadata edits drop the quality anchors while keeping the role rules (no fabrication), the format spec, and the distilled judgment layer — the fragment must still be stylistically correct and compilable. Whole-document generation does not take this tier; its anchors stay. Each scoped call saves about 2k tokens. The full design of layered prompt assembly is the subject of Part 2.

Layer 3: material trimming and stable prefixes

The material layer does two things, and the second matters more than the first.

Deterministic trimming. select_material filters at zero token cost: pure-number matrix lines (bare CSV) are dropped; lines with lexical overlap with the target section or the feedback, or “data lines with textual annotation” (e.g. “measured 0.5 mm”), are kept, capped at 2,000 characters. The filter has a floor — it never filters the context away entirely:

# keep >=2 lines or >=40 chars; otherwise treat filtering as failed
# and fall back to the first `cap` chars
return out if (len(keep) >= 2 or len(out) >= 40) else m[:cap]

The whole-document revision path gets the same treatment (triggered only above 12k characters; never on first-version generation): autonomous exploration used to write up to 120k characters of raw files back into the material, after which every chat / whole-edit turn carried 40–70k tokens — even when the feedback was just “make the tone more formal”. After filtering: 8–15k per turn.

Stable prefixes. A deep audit produced an honest conclusion: on providers with prefix caching, a slimmer system prompt saves roughly nothing in direct cost (its value is focus — keeping irrelevant text from interfering); the real money is in the material block that misses the cache every turn. DeepSeek’s context cache is fully automatic, and the hit price is about 0.8% of a miss (v4-pro: hit $0.003625/M vs miss $0.435/M — vendor list prices verified online, 2026-07). To collect that discount, the material message must be byte-identical across turns — so the trimming and per-file digest functions depend only on the material itself, never on the current turn’s feedback, and the result is pinned as a fixed messages[0] prefix. Measured input cost: about -49%, multiplicative with the relevance filtering above.

The Anthropic path requires declaring the cache explicitly, and the TTL is a genuine trade-off:

system=[{"type": "text", "text": system,
         "cache_control": {"type": "ephemeral", "ttl": "1h"}}],

The reasoning is written in the source comment: users revise documents at human pace, so a 5-minute TTL expires between turns — you pay the cache-write premium every turn and never hit. The 1h tier writes at 2× price, reads at 0.1×, and every hit renews the TTL, matching the human rhythm. Below the model’s minimum prefix threshold it silently skips caching — harmless. Measured: Claude-path input cost down 54–65%.

Layer 4: per-step model tiering

Pipeline steps are not uniformly hard: the exploration tool loop (reading files, deciding what to read next), the material health check (JSON classification), BibTeX generation, and the English-abstract translation are simple, high-frequency tasks; drafting and reflective revision are what need the flagship model. So the model is passed per step: simple steps go to the cheap tier (DeepSeek V4 Flash — input $0.14/M, output $0.28/M, about 1/3 of Pro’s unit price; the vendor claims Flash matches Pro on simple agent tasks at roughly 12× lower cost — vendor claim, not independently verified), while drafting, reflection, figure generation, and compile self-repair stay on Pro. With no cheap tier configured, everything falls back to the main model — zero behavior change; and cost accounting accrues per actual model of each call, because a single flat rate misprices runs that mix the two tiers.

The measured ledger

Typical inputs after layering context by problem type:

The opening example: ~6–7k input tokens before → 0 (fast route hits the keywords class) + ~1.5k after. The whole-document revision path drops from 40–70k to 8–15k per turn, then multiplies with the ~-49% prefix-cache discount. Compile self-repair was also switched to a fresh minimal context (current draft + error excerpt + template rules), saving 30–50k tokens per repair round.

Observation ships with the optimization: every call collects real token usage via usage_out (including cache-hit counts) into the process log and trace, so agent-lab can show “which step spent how much”. This round of changes added 10 backend tests (backend 290 passed, agent-lab 15 passed). One negative lesson: the accounting was once wrong — cost estimation did not discount cache hits, so autonomous-run costs displayed up to several times too high. Fix the observability first, or the optimization aims at the wrong target.

When not to do this

1. Never filter material for first-version generation. The material is the sole source of facts — err on the side of completeness; relevance filtering is for revision turns only.
2. Never slim the system prompt for whole-document generation. The quality anchors are where depth and style come from; that spend is worth it.
3. Never economize on the verification layer. Provenance checking still runs against the complete material server-side — you save on what is fed to the model, not on validation.
4. Nominal savings ≠ real savings. Once caching hits, many “big” optimizations shrink: splitting the exploration/figure contexts looked like 50k+ tokens saved, but the cache already absorbed about 90% of it, leaving 10–20% real savings — we downgraded that item accordingly. Rank optimizations by post-cache-discount numbers.
5. Fix observation before optimizing. When accounting ignores cache hits, the most expensive path looks even more expensive than it is, and steers the optimization effort in the wrong direction.