# The token economics of LLM applications

> Cost is the first real architectural constraint an LLM application hits. Starting from a precise ledger of a 'fix the keywords' request that shipped 6–7k input tokens, this article presents four layers of remedies — zero-token deterministic pre-routing, tiered system prompts, deterministic material trimming with stable prefixes for provider prompt caching, and per-step model tiering — plus measured results and an honest conclusion about cache economics: a slimmer system prompt saves roughly nothing once caching hits; the real money is in the material block that misses every turn.

- Canonical (HTML): https://kaguc.com/blog/token-economics/
- Date: 2026-07-29


*Also in [中文](/blog/token-economics-zh/). Part 1 of the Engineering LLM Applications series. The series is grounded in the implementation and measurements of a production-grade AI writing agent (FastAPI + React + Tauri), and discusses transferable engineering methods.*

## The problem: why does fixing one keyword cost 6–7k tokens

When you build an LLM application, the first constraint that forces an architectural decision is usually not model capability — it is the bill. Our project (a LaTeX lab-report writing agent) received a blunt question from a user: "why did you send 6k?" — he only wanted to change the report's keywords.

The precise ledger (before the fix):

| Component | Content | Size (approx.) |
|---|---|---|
| System prompt | Role rules + report spec + writing methodology (1,948 chars) + golden-sample exemplar (1,235 chars) + LaTeX template rules | ~3k tokens |
| Session material | First 6,000 chars of `material` — including the entire experiment txt bundle (data matrices, field-sweep parameters, …) | ~3–4k tokens |
| Target block + instruction | One `\keyword{...}` line + the edit request | ~0.1k |
| Total | | ~6–7k in / ~50 out |

The root cause is not the model but one-size-fits-all context assembly: whatever the edit, the full set of quality anchors plus the full material gets stuffed in. Fixing a keyword has no use for a complete exemplar report about a different experiment, and even less for raw data matrices. The user's follow-up became the design charter for this pipeline: "the process may be complex, but the goal is to save tokens." Cost optimization is not cutting features; it is layering context by problem type — minimal context per type. Below are the four layers we shipped.

## Layer 1: zero-token deterministic pre-routing

"Which part does this feedback want to change" is a routing problem, and the default is to ask a model. But a substantial share of feedback carries an unambiguous explicit target: bibliography, abstract, keywords, "Table 2", "Figure 3", "§3.1", a uniquely matching section title. These are decided with regexes and string matching (`_fast_route` — zero LLM calls, no API key even needed): exactly one signal class hit → route directly; "whole document" wording, no signal, or conflicting classes → fall back conservatively to LLM routing, never guess.

The gain is more than one saved routing call: the decision has zero latency, is 100% explainable (the log is tagged "deterministic fast decision, 0 tokens"), and offline tests run at full speed. This pattern is developed further as part of the deterministic-boundary principle in [Part 11](/blog/deterministic-boundary/).

## Layer 2: tiered system prompts

The most expensive part of the system prompt is the quality anchors: the writing methodology plus the golden-sample exemplar, about 3.2k characters, roughly 2k tokens. They are indispensable for whole-document generation — an early refactor once dropped them and generation depth and style collapsed immediately — but useless for small edits like "fix one section" or "fix the keywords".

So `build_system(scoped=True)` adds a tier: paragraph-level and metadata edits drop the quality anchors while keeping the role rules (no fabrication), the format spec, and the distilled judgment layer — the fragment must still be stylistically correct and compilable. Whole-document generation does not take this tier; its anchors stay. Each scoped call saves about 2k tokens. The full design of layered prompt assembly is the subject of [Part 2](/blog/prompt-assembly/).

## Layer 3: material trimming and stable prefixes

The material layer does two things, and the second matters more than the first.

**Deterministic trimming.** `select_material` filters at zero token cost: pure-number matrix lines (bare CSV) are dropped; lines with lexical overlap with the target section or the feedback, or "data lines with textual annotation" (e.g. "measured 0.5 mm"), are kept, capped at 2,000 characters. The filter has a floor — it never filters the context away entirely:

```python
# keep >=2 lines or >=40 chars; otherwise treat filtering as failed
# and fall back to the first `cap` chars
return out if (len(keep) >= 2 or len(out) >= 40) else m[:cap]
```

The whole-document revision path gets the same treatment (triggered only above 12k characters; never on first-version generation): autonomous exploration used to write up to 120k characters of raw files back into the material, after which every chat / whole-edit turn carried 40–70k tokens — even when the feedback was just "make the tone more formal". After filtering: 8–15k per turn.

**Stable prefixes.** A deep audit produced an honest conclusion: on providers with prefix caching, a slimmer system prompt saves roughly nothing in direct cost (its value is focus — keeping irrelevant text from interfering); the real money is in the material block that misses the cache every turn. DeepSeek's context cache is fully automatic, and the hit price is about 0.8% of a miss (v4-pro: hit $0.003625/M vs miss $0.435/M — vendor list prices verified online, 2026-07). To collect that discount, the material message must be byte-identical across turns — so the trimming and per-file digest functions depend only on the material itself, never on the current turn's feedback, and the result is pinned as a fixed `messages[0]` prefix. Measured input cost: about -49%, multiplicative with the relevance filtering above.

The Anthropic path requires declaring the cache explicitly, and the TTL is a genuine trade-off:

```python
system=[{"type": "text", "text": system,
         "cache_control": {"type": "ephemeral", "ttl": "1h"}}],
```

The reasoning is written in the source comment: users revise documents at human pace, so a 5-minute TTL expires between turns — you pay the cache-write premium every turn and never hit. The 1h tier writes at 2× price, reads at 0.1×, and every hit renews the TTL, matching the human rhythm. Below the model's minimum prefix threshold it silently skips caching — harmless. Measured: Claude-path input cost down 54–65%.

## Layer 4: per-step model tiering

Pipeline steps are not uniformly hard: the exploration tool loop (reading files, deciding what to read next), the material health check (JSON classification), BibTeX generation, and the English-abstract translation are simple, high-frequency tasks; drafting and reflective revision are what need the flagship model. So the model is passed per step: simple steps go to the cheap tier (DeepSeek V4 Flash — input $0.14/M, output $0.28/M, about 1/3 of Pro's unit price; the vendor claims Flash matches Pro on simple agent tasks at roughly 12× lower cost — vendor claim, not independently verified), while drafting, reflection, figure generation, and compile self-repair stay on Pro. With no cheap tier configured, everything falls back to the main model — zero behavior change; and cost accounting accrues per actual model of each call, because a single flat rate misprices runs that mix the two tiers.

## The measured ledger

Typical inputs after layering context by problem type:

| Problem type | System prompt | Material | Typical in |
|---|---|---|---|
| Routing (explicit target) | — | — | 0 (deterministic decision) |
| Routing (everything else) | Small router system | none | ~1.5–2.5k |
| Edit abstract/keywords | Scoped tier | none (document outline digest instead) | ~1.5k |
| Edit one body section | Scoped tier | relevance-filtered, ≤2,000 chars | ~2–3k |
| Whole-document rewrite | Full (quality anchors intact) | full | 10k+ (money well spent) |

The opening example: ~6–7k input tokens before → 0 (fast route hits the keywords class) + ~1.5k after. The whole-document revision path drops from 40–70k to 8–15k per turn, then multiplies with the ~-49% prefix-cache discount. Compile self-repair was also switched to a fresh minimal context (current draft + error excerpt + template rules), saving 30–50k tokens per repair round.

Observation ships with the optimization: every call collects real token usage via `usage_out` (including cache-hit counts) into the process log and trace, so agent-lab can show "which step spent how much". This round of changes added 10 backend tests (backend 290 passed, agent-lab 15 passed). One negative lesson: the accounting was once wrong — cost estimation did not discount cache hits, so autonomous-run costs displayed up to several times too high. Fix the observability first, or the optimization aims at the wrong target.

## When not to do this

1. **Never filter material for first-version generation.** The material is the sole source of facts — err on the side of completeness; relevance filtering is for revision turns only.
2. **Never slim the system prompt for whole-document generation.** The quality anchors are where depth and style come from; that spend is worth it.
3. **Never economize on the verification layer.** Provenance checking still runs against the complete material server-side — you save on what is fed to the model, not on validation.
4. **Nominal savings ≠ real savings.** Once caching hits, many "big" optimizations shrink: splitting the exploration/figure contexts looked like 50k+ tokens saved, but the cache already absorbed about 90% of it, leaving 10–20% real savings — we downgraded that item accordingly. Rank optimizations by post-cache-discount numbers.
5. **Fix observation before optimizing.** When accounting ignores cache hits, the most expensive path looks even more expensive than it is, and steers the optimization effort in the wrong direction.