# Order reliability: idempotency, unknown states, and reconciliation

> Exactly-once delivery does not exist. For an irreversible action like an order, that impossibility forces a specific design: a key minted before you send, an explicit unknown state, and reconciliation instead of retries.

- Canonical (HTML): https://kaguc.com/blog/order-reliability/
- Date: 2026-08-28


*Also in [中文](/blog/order-reliability-zh/). Part 2 of [Trading app architecture](/blog/trading-app-architecture/).*

You tap submit. The request goes out. Nothing comes back.

That is the entire problem, and it has no clean solution — only a choice of which failure you are willing to own. A client that ignores this ships a bug that only appears when the network is bad, which is to say, in exactly the conditions where the money matters most.

## The impossibility you must design around

When a request times out, two situations are indistinguishable from the client: the server never received it, or the server executed it and the acknowledgement was lost. There is no message you can send, and no timeout you can tune, that separates those cases. The information does not exist on your side.

So you choose:

**At-most-once.** Never retry. You never place a duplicate order, and you sometimes silently fail to place one the user asked for. For money, unacceptable — a dropped order is invisible, and the user finds out from the market.

**At-least-once.** Always retry until acknowledged. You never lose an order, and you sometimes place it twice. Also unacceptable on its own, but this one is *fixable*.

You take at-least-once and remove the duplicate with an **idempotency key**: an identifier the client mints for the *intent*, which the server uses to recognize a repeat. First arrival executes. Every subsequent arrival with that key returns the original result without executing again. The retry becomes safe by construction, and "exactly once" is achieved not at the transport layer, where it is impossible, but at the **semantic** layer, where it is merely careful bookkeeping.

## Mint the key before you send, and persist it

This is the detail that separates a design that works from one that looks identical and doesn't.

The key must be generated and **written to durable storage before the first network call**, as part of creating the order intent. Not inside the request builder. Not as a retry-time afterthought.

The reason is the failure you are protecting against. If the app is killed mid-flight, then on relaunch you must be able to ask "did that order go through?" — and you can only ask if the identity of the attempt survived the crash. A key generated in memory at send time dies with the process, and your recovery path invents a *new* key, which the server correctly treats as a *new order*. You have implemented duplicate submission with extra steps.

So the sequence is: create the intent, persist it with its key and a pending status, then send. Every retry, in this process or a later one, reuses that key.

## Timeout is a state, not a failure

The most common modeling error is a boolean: submitted or not. Reality has a third value.

Model the order as a state machine with an explicit **unknown** state, entered on timeout or ambiguous error. Unknown means: an action may or may not have happened, and the client is not entitled to guess.

What you do in unknown is the important part: **you do not blindly retry, and you do not report failure.** You *ask*. Query order status by the idempotency key, and let the authoritative answer collapse the ambiguity. Only if the server confirms it never saw the request do you resend.

Blind retry is safe only if your idempotency guarantee is airtight end to end, including across app restarts and server deploys. Query-then-decide is safe regardless. On an irreversible action, prefer the design whose safety doesn't depend on a guarantee elsewhere being perfect.

Telling the user "failed" on a timeout is the same class of error in the opposite direction: they resubmit by hand, and now the duplicate is caused by your UI rather than your network layer. An honest pending state is not a worse user experience. It is the only truthful one.

## The server is the only source of truth

A local "sent" is not "accepted." "Accepted" is not "filled." Only the endpoint that owns the money can make those statements, and this is the end-to-end principle applied to a domain where it has teeth.

Two consequences for the client:

**Optimistic UI must be labeled as optimistic.** Showing a pending order immediately is good; showing it as *placed* is a lie you will occasionally have to retract in front of someone who acted on it.

**Reconcile on every reconnect and every cold start.** Pull the authoritative order list and let it overwrite local state. Do not merge cleverly. The client's copy is a cache, and a cache that argues with the origin is just a bug with persistence.

## Ordering: version, don't trust arrival

Status updates for one order can arrive out of order — retries, multiple channels, reconnects. If you apply whatever arrives last, a stale "submitted" can overwrite a fresh "filled," and your UI now shows a completed order as in-flight.

Give every order a monotonically increasing version or sequence assigned **by the server**, and drop any update whose version is not newer than what you hold. Last-write-wins by version, never by arrival time. Client clocks are not a substitute; they are wrong, and they are wrong in both directions.

Make illegal transitions loud, too. If you receive `filled → submitting`, that is not a state to render, it is a defect to log. Silently accepting impossible transitions is how a data bug becomes a customer's screenshot.

## Store events, not just current state

Keep an append-only record of what happened: intent created, submitted, acknowledged, filled. Derive the current state from it.

You want this for three reasons, and only one of them is regulatory. It gives support and engineering a truthful answer to "what did the app actually do at 09:31:04." It makes reconciliation a comparison rather than a guess. And it turns your postmortems from archaeology into reading.

Mutating a single status field in place destroys exactly the information you will need on the one day you need it badly.

## Scaling this down

All of the above is the price of **irreversibility**. Take that away and it collapses.

If the action is reversible — a saved draft, a like, a preference — a plain request with a retry is correct, and an idempotency key plus a state machine plus an event log plus reconciliation is machinery in search of a problem.

The trigger for adopting this design is not "we handle data," it's "**an action that cannot be undone by the user.**" Payments, orders, transfers, irreversible account changes. When you have one, build all of it. When you don't, don't.


## References

- E. A. Akkoyunlu, K. Ekanadham, R. V. Huber, ["Some constraints and tradeoffs in the design of network communications"](https://doi.org/10.1145/800213.806523), *SOSP* 1975 — the Two Generals problem: agreement over an unreliable channel is impossible, so exactly-once delivery is too.
- P. Helland, ["Idempotence Is Not a Medical Condition"](https://queue.acm.org/detail.cfm?id=2187821), *ACM Queue* 10(4), 2012 — at-least-once plus idempotent handling, and why a timeout must be modeled as an unknown state.
- J. H. Saltzer, D. P. Reed, D. D. Clark, ["End-to-End Arguments in System Design"](https://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf), *ACM TOCS* 2(4), 1984 — a guarantee is only real at the endpoint that can verify it.
- L. Lamport, ["Time, Clocks, and the Ordering of Events in a Distributed System"](https://lamport.azurewebsites.net/pubs/time-clocks.pdf), *CACM* 21(7), 1978 — order by logical version, never by wall clock or arrival.
- Stripe, [Idempotent requests](https://docs.stripe.com/api/idempotent_requests), and IETF HTTPAPI WG, ["The Idempotency-Key HTTP Header Field"](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) — the production shape of client-minted idempotency keys.
- M. Fowler, ["Event Sourcing"](https://martinfowler.com/eaaDev/EventSourcing.html), 2005 — derive current state from an append-only record of what happened.

---

*Previous: [Real-time data pipelines](/blog/realtime-data-pipeline/) · Next: [Local-first data and delta sync](/blog/local-first-sync/)*