August 13, 2026 · 6 min read
Replication gets interesting when the copies disagree
My notes on leaders, lag, conflicts, and quorums, with small demos you can break on purpose.
- system-design
- databases
- distributed-systems
These are my notes from chapter 5 of Martin Kleppmann’s Designing Data-Intensive Applications. The chapter changed the way I think about replication. Keeping two copies of a value is easy; deciding what the system should do while those copies disagree is the real design problem.
We replicate data to put it closer to readers, survive machine failures, and spread traffic around. All three sound straightforward until a write arrives.
Start with one writer
The simplest useful setup has a leader and one or more followers. Every write goes to the leader. The leader records it in a log, and the followers replay that log in order. Reads may go to either place; writes have one door.
The leader answers the client first and ships the write afterwards. Fast, but followers trail behind and can serve stale reads.
NYC
leaderus-east
0
v0
London
followereu-west
0
v0
Tokyo
followerap-northeast
0
v0
Run an operation above and every step lands here — scrub, replay, or slow it down.
Pick how the leader acknowledges, write a likes count, then read each replica to see who has caught up.
With synchronous replication, the leader waits for a follower before telling the client the write succeeded. That adds latency, but there is already another durable copy. With asynchronous replication, the leader answers first and followers catch up later. It feels faster right up until the leader dies with an acknowledged write that no one else received.
Try an async write in the demo, then crash the leader. The replacement can only recover what reached a surviving node. The missing write does not become less missing because the client once saw “success.”
Failover has a few jobs: notice the leader is gone, choose a follower that is caught up enough, redirect clients, and make the remaining followers use the replacement. The nightmare case is split-brain, where two machines both accept writes as leader and produce histories that cannot be cleanly joined.
What followers actually receive
A follower does not periodically copy the whole database. It consumes a stream of changes. That stream can take several forms:
| Stream | What travels | Typical snag |
|---|---|---|
| Statement-based | The original SQL (UPDATE likes SET n = n + 1) | Non-determinism: NOW(), RAND(), triggers |
| Write-ahead log (WAL) | The leader’s disk-level bytes | Tied to storage version; painful for online upgrades |
| Logical / row-based | A sequence of row changes | More work to decode; much easier to reason about |
Logical row changes are often the practical middle ground. They describe the effect of a write without tying every follower to the leader’s exact storage-engine version.
Lag leaks into the UI
Once a follower is allowed to run behind, “read from a replica” no longer means “show me the latest value.” Users see the difference:
- You save a profile, refresh, and your change vanishes for a moment. That breaks read-your-writes.
- One request shows a new value and the next request, routed to a slower replica, shows the old one. That breaks monotonic reads.
- A reply appears before the message it answers. That breaks a consistent prefix of causally related writes.
The leader answers the client first and ships the write afterwards. Fast, but followers trail behind and can serve stale reads.
NYC
leaderus-east
0
v0
London
followereu-west
0
v0
Tokyo
followerap-northeast
0
v0
Run an operation above and every step lands here — scrub, replay, or slow it down.
Write, then immediately read a follower. Async replication makes read-your-writes fail unless you pin the session to the leader.
There is no magic switch for this. You can route a user’s reads to the leader for a while, remember the log position of their last write and choose a replica that has passed it, or synchronously replicate the writes where loss would hurt most. Each option moves the cost somewhere else.
More leaders, more arguments
One leader becomes awkward when users and datacenters are far apart. If London must send every write to Virginia, distance is part of every request. A multi-leader setup lets each region accept local writes and exchange them later.
Now two people can update the same record without either region seeing the other update. Last-write-wins resolves the conflict by deleting someone’s work. Sometimes that is acceptable; often it is just easy. A merge based on the data—a set union for a cart, for example—can preserve both changes.
Keep the newer timestamp and drop the other cart. Simple, and it silently loses one shopper's item.
London
leadereu-west
bread
v1
NYC
leaderus-east
bread
v1
Run an operation above and every step lands here — scrub, replay, or slow it down.
Both datacenters start with bread. Partition them, add milk in NYC and eggs in London, then heal the link.
This model makes sense for independently operating regions or clients that spend time offline, such as calendars and point-of-sale systems. I would not choose it just to avoid the phrase “single leader.” Conflict handling becomes part of the product.
No leader: ask several replicas
Dynamo-style systems take another route. Writes go to several replicas, reads query several replicas, and a coordinator waits for enough answers.
With N replicas, let W be the number of write acknowledgements and R the number of read responses. The familiar rule is:
W + R > N
Under simplified conditions, that makes every read set overlap every successful write set. It means the latest write is available to the read; it does not guarantee the system will recognize and choose it correctly. You still need versions and a conflict rule. The demo deliberately writes to one end of the replica list and reads from the other, so you can see exactly when the sets stop overlapping.
Next write W+R = 4 ≤ N=5 — write set n1, n2 and read set n4, n5 can be disjoint.
n1
replicarack-1
—
v0
n2
replicarack-2
—
v0
n3
replicarack-3
—
v0
n4
replicarack-4
—
v0
n5
replicarack-5
—
v0
Run an operation above and every step lands here — scrub, replay, or slow it down.
Writes go to the first W nodes; reads come from the last R. Sets overlap only when W+R > N.
Real systems complicate the neat equation. A sloppy quorum may write to any reachable node instead of the intended replicas. Hinted handoff and anti-entropy move those writes home later. This keeps the service available during a partition, but it also creates repair work for the future.
Quorums do not quietly provide transactions, uniqueness, or a single agreed history. They give you knobs for latency, durability, and stale-read risk.
How I keep the options straight
| Setup | Writes | Conflicts | Failure story |
|---|---|---|---|
| Single leader, sync replica | One node | None | Leader wait; durable spare copy |
| Single leader, async replicas | One node | None (until failover) | Fast; newest writes can vanish |
| Multi-leader | Many regions | Yes — you must merge | Each site stays writable |
| Leaderless quorum | Any W nodes | Concurrent writes need version merge | No failover dance; tune W/R |
In the first demo, try this sequence: write asynchronously, read from London, then crash the leader. It packs most of the chapter into three clicks. A follower that may lag can return old data. A follower that has not caught up can only become leader by leaving acknowledged writes behind.
Synchronous replication, sticky sessions, and W + R > N look like separate techniques, but they all answer the same question: which completed writes is this read allowed to miss?