# Real-time data pipelines: push, backpressure, and conflation

> A price feed is not a fast API. It is an unbounded stream whose consumer is slower than its producer, which makes backpressure the central design decision — and the answer depends on whether your stream is a log or a register.

- Canonical (HTML): https://kaguc.com/blog/realtime-data-pipeline/
- Date: 2026-08-28


*Also in [中文](/blog/realtime-data-pipeline-zh/). Part 1 of [Trading app architecture](/blog/trading-app-architecture/).*

Most client code assumes a shape: you ask, the server answers, you render. A market data feed breaks that assumption in three places at once. Nobody asks — the data arrives on its own. There is no "answer" — the stream never ends. And the producer is faster than the consumer, permanently, by design.

Get those three straight and the architecture of a real-time client mostly writes itself.

## Why the data must be pushed

The naive approach is to poll: ask every *n* milliseconds. It fails on arithmetic before it fails in production.

Polling puts a floor under your staleness equal to the interval. Want data no more than 100 ms old? Poll ten times a second. Now multiply: every symbol on screen, every open client. The server does full request-handling work on every tick regardless of whether anything changed — and most of the time nothing has. You are paying for the *clock*, not for the *information*.

Push inverts the cost. Work happens when the data changes. An idle market costs an idle connection; a volatile one costs exactly the messages that carry news. That is the property you want: **cost proportional to change, not to time.**

The price of push is that a connection is no longer an event, it is *state you own*. That state needs:

- **Heartbeats**, because TCP will happily hold a connection that is functionally dead. Without an application-level ping you cannot distinguish "quiet market" from "we were severed twenty minutes ago." Silence must be made falsifiable.
- **Reconnect with exponential backoff and jitter.** Backoff so you don't hammer a struggling server; jitter because without it every client that dropped at the same moment returns at the same moment, and your recovery becomes a second outage.
- **A resume token**, so reconnecting doesn't mean starting from nothing.

## Gaps, and the difference between a log and a register

Every message carries a sequence number. Not for bookkeeping — for detecting the messages you *didn't* get. A silently dropped update in a price feed isn't missing data, it's **wrong data**: your client is now displaying a number it believes is current and isn't.

What you do about a gap depends on what kind of stream it is, and this is the classification that drives everything downstream:

A **register** holds a current value. The last write wins; earlier writes are irrelevant once superseded. A last-traded price is a register — if you missed three ticks and the fourth arrives, you have lost nothing that the fourth doesn't already tell you.

A **log** is a sequence of deltas where each one is meaningful. An order book maintained by incremental updates is a log — miss one and every subsequent update is applied to a corrupt base. The damage is permanent and silent; it compounds.

So: for a register, a gap is a hiccup. For a log, a gap is a **resync** — discard local state, request a fresh snapshot, apply increments from that snapshot's sequence forward. The universal pattern is **snapshot plus increments, with the snapshot's sequence number as the join point.**

Write down, for each stream in your product, whether it is a log or a register. It is a five-minute exercise that decides the next section for you.

## Backpressure is the real design problem

Your consumer is a phone. Your producer is a market. Under load the producer wins, and "wins" means your queue grows without bound until the process dies.

There are only three honest responses:

**Buffer everything.** Correct in principle, fatal in practice — memory grows without limit, and worse, the data you eventually render is old. You built a queue that converts a freshness problem into a latency problem and then into a crash.

**Drop.** Shed messages when behind. Legitimate for a register, catastrophic for a log.

**Conflate.** Keep only the newest value per key, coalescing anything that arrives before you can consume it. Ten ticks for a symbol between two frames collapse into one. This is the right answer for registers, and it is not a degradation: rendering the latest price is *exactly correct*, because the intermediate ticks were already superseded before a human could perceive them.

Conflation is the single highest-leverage idea in a real-time client, and it follows directly from the register/log split. You cannot conflate a log; you must never buffer a register.

## Decouple tick rate from frame rate

A screen refreshes 60 times a second. A liquid symbol can update far more often than that. So updates-per-second above 60 have **no representable meaning on the client** — they can only cost you.

The architecture that follows: an ingest path that decodes and conflates into a per-symbol current-value map, and a render path that samples that map once per frame. Ticks and frames are now independent. A tenfold burst in market activity changes the ingest cost slightly and the render cost not at all.

The common failure is wiring the socket directly to UI state, so every message triggers a re-render. That design works beautifully in testing and collapses on the first volatile open, because you accidentally coupled your frame rate to the market's.

## Where the native boundary belongs

If you cross a language boundary (Dart to C++, JVM to native), the cost is mostly **per call**, not per byte. That single fact places the boundary for you.

Put decode, gap detection, conflation, and fan-out on the fast side. Cross once per frame with a batch of changed symbols. Do not cross per message.

A client that calls across the boundary for every tick can be slower than one that does everything in the managed language, because it pays thousands of transition costs to save a decode. The reason to go native here is not that "C++ is fast" — it's that a managed runtime's *tail* latency is unpredictable. A garbage collection pause is invisible in your average and clearly visible as a stutter at the worst moment. You go native for **timing determinism** on the hot path, and only for that.

## Subscriptions are a lifecycle, not a call

The last piece is bookkeeping that is easy to underestimate:

- **Reference-count subscriptions per symbol.** Two widgets showing the same symbol should produce one subscription, and it should end when the second one leaves.
- **Resubscribe on reconnect.** Server-side subscription state died with the socket. If your client assumes otherwise, it shows a frozen screen that looks exactly like a quiet market — the worst possible failure, because it is indistinguishable from working.
- **Degrade in the background.** An app that isn't visible should unsubscribe or drop to a slow summary. This is the difference between a well-behaved app and one users uninstall for battery reasons.

## When you need none of this

Everything above is forced by one number: how often your data changes relative to how fast a human perceives it. If a value updates every few seconds, a WebSocket handing JSON straight to your UI state is the *correct* design, and every mechanism above is waste.

The discipline is to add each piece when a measurement demands it, in roughly this order: push before polling hurts; sequencing when correctness demands it; conflation when the queue grows; a native boundary only when profiling shows the decode loop, and never before.


## References

- J. Kreps, ["The Log: What every software engineer should know about real-time data's unifying abstraction"](https://web.archive.org/web/20190913020020/https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying), LinkedIn Engineering, 2013 (archived) — the log as an append-only sequence where every entry matters, and snapshot + offset as the replication primitive.
- [Reactive Streams specification](https://www.reactive-streams.org/), 2015 — backpressure as a first-class contract between producer and consumer; the three honest responses (buffer, drop, conflate) are what a consumer can do when it has none.
- M. Brooker, ["Exponential Backoff and Jitter"](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/), AWS Architecture Blog, 2015 — why backoff without jitter turns recovery into a second outage.
- J. Dean, L. A. Barroso, ["The Tail at Scale"](https://research.google/pubs/the-tail-at-scale/), *CACM* 56(2), 2013 — why percentiles, not averages, describe what a user experiences.
- Binance, [WebSocket Streams — "How to manage a local order book correctly"](https://developers.binance.com/docs/binance-spot-api-docs/web-socket-streams) — a public specification of snapshot + increments joined at a sequence number.

---

*Next: [Order reliability](/blog/order-reliability/) — why exactly-once doesn't exist, and what to do instead.*