Eventual Consistency & Convergence | Microservices Course

Problem

Even causal consistency constrains the write path: dependencies must be tracked and a write must wait for its causes before becoming visible. Some systems can't accept any constraint on writes at all. They have to take a write during a network partition, on a continent with no nearby quorum, or on a phone that's offline, and they can't wait for a leader or a majority to respond. For these systems the cost of rejecting a write (a shopping cart that won't accept an item, a name server that won't answer) outweighs the cost of a briefly stale or conflicting read.

So you let every replica accept writes locally and diverge from its peers, which raises the real problem: once two replicas have independently accepted conflicting writes, something has to bring them back to the same value, or they stay divergent forever.

Solution

Let every replica serve reads and writes without coordinating, propagate updates to peers asynchronously, and define a deterministic merge rule so that once propagation catches up, every replica holds the same value. The only guarantee is convergence: if writes stop, replicas eventually agree. The model says nothing about how long divergence lasts or what an intermediate read returns, so you design around the divergence window rather than trying to eliminate it.

The design decision that matters is how concurrent conflicting writes reconcile:

Last-writer-wins — timestamp every write, keep the highest. Simple, but silently discards the losing write and depends on clock discipline; skew causes lost updates.
Version vectors — detect when two writes are concurrent rather than ordered, and surface both as siblings for the application to merge. Preserves data at the cost of pushing resolution up the stack.
CRDTs — data types whose merge is commutative, associative, and idempotent, so concurrent updates converge deterministically with no coordination (counters, sets, maps).

Divergence is actively closed by anti-entropy: read repair fixes stale replicas when a read notices disagreement, hinted handoff replays writes a down node missed, and background Merkle-tree comparison finds and repairs differences between replicas. Dynamo-style systems also let you tune the read and write quorum sizes against the replica count to trade availability for freshness.

Tradeoffs

Property	Effect
Availability and latency	Reads and writes never block on coordination and survive partitions; the reason to choose this
Staleness	A read can return an old value, with no recency bound
Conflict handling	Concurrent writes must be reconciled: last-writer-wins loses data, version vectors offload merging to the app, CRDTs constrain the data model
Divergence window	Unbounded in theory, bounded by propagation speed in practice; correctness requires assuming reads may be stale or conflicting
Ordering	Basic eventual consistency gives no ordering across keys and isn't even causal by default; session or causal guarantees are layered on top
Operational cost	Read repair, hinted handoff, and Merkle sync consume background bandwidth and tuning effort

Implementations

Minimal pseudocode

def write(key, value):
    incoming = (value, clock.now())
    store[key] = merge(store.get(key), incoming)
    gossip(key, store[key])               # async, best-effort
    return OK                             # ack without waiting for peers

def on_gossip(key, incoming):
    store[key] = merge(store.get(key), incoming)

def merge(a, b):                          # must be commutative + idempotent
    if a is None: return b
    return a if a.ts >= b.ts else b       # last-writer-wins example

Convergence depends entirely on merge being commutative, associative, and idempotent, so replicas reach the same result regardless of message order or duplication.

DynamoDB

Descends from Amazon's Dynamo design (2007): consistent hashing, asynchronous replication, hinted handoff, and Merkle-tree anti-entropy, with the original paper resolving conflicts through vector clocks and application-level merge such as the union of a shopping cart. The managed DynamoDB service defaults to eventually consistent reads and lets you request a strongly consistent read per call; its cross-region global tables reconcile concurrent writes with last-writer-wins rather than exposing siblings.

Cassandra

Dynamo-influenced, with consistency tunable per query (ONE, QUORUM, ALL) so each operation chooses its own point on the availability/freshness curve. Conflicts are resolved by last-writer-wins at the individual cell using write timestamps, which means concurrent writes to the same cell can lose data. Divergence is repaired through read repair, hinted handoff, and Merkle-tree repair runs.

DNS

The everyday eventual-consistency system. A change at an authoritative server propagates to caching resolvers, and during a record's TTL a resolver keeps serving the old value, so different clients see different answers for a while. Once TTLs expire and caches refetch, every resolver converges on the new record. The convergence rule is simply that an expired cache entry is replaced by a fresh fetch.

Riak

The closest production system to the Dynamo paper. It uses dotted version vectors to detect concurrent writes and can either return siblings for the application to resolve or fall back to last-writer-wins. It also ships convergent CRDTs (counters, sets, maps, registers, flags) that merge automatically without surfacing conflicts, and it closes divergence with read repair plus active anti-entropy built on Merkle trees.