Money, Identity & Correctness-Critical

Inventory / Reservation System

Sell limited stock under heavy contention without overselling: an atomic conditional decrement, idempotency keys for safe retries, time-bounded leases for checkout holds, and a reconciler that reclaims stranded inventory.

~30 min · advanced

Problem & Requirements

Sell a finite number of things — concert seats, a limited SKU — to many buyers arriving at once, and never sell the same unit twice. A buyer reserves an item, takes some time to pay, and either confirms or walks away. Reservations vastly outnumber the stock during a spike, the same buyer may retry after a timeout or a double-click, and a buyer who abandons a half-finished checkout must not hold inventory forever. The system is correctness-critical: overselling a unique seat is a refund, an apology, and a reputation hit, not a rounding error.

Correctness here cannot come from reading the count and then writing it back, because two buyers who both read "1 left" will both decrement and both win. It comes from a single atomic conditional write — decrement only if stock remains — so exactly one of two concurrent buyers succeeds and the other is told no. This is optimistic concurrency, and it is the spine: no lock is held across a buyer's think-time, and the conditional decrement is the one place overselling is prevented. Around that core, idempotency keys make a retried request apply once, leases turn a reservation into a hold that expires if not paid, and eventual reconciliation sweeps up holds that were never confirmed so stock doesn't leak. Every other decision exists to protect the conditional decrement from retries, abandonment, and partial failure.

Functional

reserve(item, idem_key) atomically holds one unit if any remain; confirm(hold, idem_key) converts a live hold into a sale.
A reservation that is not confirmed within its window expires and returns its unit to stock.
A retried request with the same idempotency key applies once and returns the same result.

Non-functional (back-of-envelope)

Quantity	Target	What it forces
Contention	thousands of buyers, one hot item	Atomic conditional write; never read-then-write
Oversell tolerance	zero for unique items	CAS on the count; the loser must fail, not retry into a double-sell
Duplicate request rate	high (timeouts, double-clicks)	Idempotency keys; same key applies once, same result returned
Hold duration	seconds to minutes (checkout window)	Leases with a TTL; abandoned holds must expire, not stick
Confirm latency	seconds, includes payment	The hold decouples reserve from pay; commit is idempotent
Reconciliation lag	minutes	A background sweep reclaims expired holds and repairs drift
Invariant	`available + held + sold == total`, always	Eventual reconciliation: transient drift tolerated, permanent drift not

The treacherous row is the invariant, not the oversell row. Preventing oversell is the headline and the conditional decrement handles it in build step two. The subtle failure is the opposite leak: a buyer holds a unit, then crashes or abandons checkout, and that unit sits in a "held" state that nothing returns to stock. No oversell ever happens, every check passes, and yet available inventory silently erodes until the item shows "sold out" while units actually sit unclaimed. Catching that is build step six, and it is the difference between a reservation system and a slow inventory leak nobody notices until revenue dips.

Design

Six components, each tied to the principle it applies:

Inventory record — the unit of contention: an available count (or a set of unique units) plus a version, against which the conditional write runs.
Optimistic reserve — a reservation is a conditional decrement that succeeds only if stock remains, so concurrent buyers can't both win. This is optimistic concurrency.
Idempotency layer — each request carries a client-supplied key; the server records the result keyed by it and replays that result on a repeat rather than re-applying. This is idempotency.
Lease / hold — the decrement moves a unit into a time-bounded hold rather than an immediate sale, and the hold lapses if not confirmed in its window. This is leases.
Confirm and commit — converting a live hold into a sale atomically and idempotently, rejecting an expired hold so the buyer restarts cleanly.
Reconciler — a background sweep that expires stale holds back to available and repairs any drift between the count and the sum of holds and sales. This is eventual reconciliation.

The two named domains differ in the unit and in their oversell stance. Ticketing reserves a unique seat, so oversell is never tolerable and each seat gets its own short hold during checkout — and because onsales spike violently, a waiting room or virtual queue usually sits in front to admit buyers at a rate the inventory can serve. E-commerce checkout reserves a quantity of a fungible SKU, often tolerates a controlled oversell that reconciliation or backorder resolves, and leans on idempotency keys end-to-end — Stripe's idempotency-key design is the canonical implementation, storing the first response and replaying it for any retry of the same key. Both use the same conditional-write core; the production counterpoint to optimistic CAS is pessimistic locking (SELECT ... FOR UPDATE), which serializes buyers on the row instead of letting them race and is simpler but throttles a hot item. The optimistic version-column pattern itself is what ORMs ship as optimistic locking (Hibernate's @Version, Rails' lock column), so the mechanism here is the same one production code already uses.

Build it

Start with the obvious version: read the available count, and if it's positive, decrement and sell. It is correct when one buyer acts at a time, and it states the contract — one unit out per successful reservation. The failure is a race between the check and the act: two concurrent buyers both read available == 1, both pass the check, and both decrement, so the item oversells. The next step closes that gap with an atomic conditional write.

Collapse check-and-decrement into one atomic conditional write. Decrement only where stock remains, in a single statement the database executes atomically, so of two concurrent buyers exactly one updates a row and the other updates none and is told no. This is optimistic concurrency: no lock is held across the buyer, the conflict is detected by the conditional WHERE, and the loser fails cleanly. The failure it exposes is retries — a buyer whose success response was lost to a timeout retries and decrements a second time, buying two.

A retried request must not apply twice. Have the client attach a unique idempotency key to the reservation, record the result keyed by it, and on any repeat of that key return the stored result without touching inventory. This is idempotency, the Stripe-style guarantee that a retry is safe. The key insert and the decrement commit in one transaction, so the operation and its record can't diverge. The failure that remains is the half-finished reservation: a buyer reserves, never pays, and the unit is decremented out of stock with nothing to ever return it.

A reservation should be a temporary claim, not a permanent removal. Move the decremented unit into a hold stamped with an expiry — the checkout window — rather than marking it sold. This is a lease: the buyer owns the unit only until the timer runs out, after which it is eligible to return to stock, so an abandoned cart stops starving inventory. The failures left over are the commit path and the cleanup: there is no way yet to turn a valid hold into a sale, and an expired or crashed hold still occupies stock until something reclaims it.

Add the commit. Confirm locks the hold, checks it is still live, and atomically flips it from held to sold — and, carrying its own idempotency key, a retried confirm after a lost response replays the same result rather than committing twice. An expired hold is rejected so the buyer restarts cleanly rather than paying for a unit that has already lapsed back to others. This combines leases and idempotency: payment runs against the live hold, and the commit is safe to retry. The remaining failure is the leak the requirements flagged — an expired or orphaned hold still subtracts from available until someone reclaims it.

Run a background sweep to make the invariant true again. First, reclaim every expired hold by deleting it and returning its unit to available, so abandoned and crashed checkouts stop leaking stock. Second, repair drift directly: for each item, recompute what available should be — total minus sold minus active holds — and correct it if a partial failure left the count wrong. This is eventual reconciliation: transient disagreement between the count and reality is tolerated for minutes, but it is never permanent, and available + held + sold == total is restored on every pass.

class Inventory:
    def __init__(self, total):
        self.available = total

    def reserve(self):
        if self.available > 0:           # check
            self.available -= 1          # ...then act — the gap oversells
            return True
        return False

Tradeoffs

Decision	What it buys	What it costs
Optimistic concurrency	No lock across think-time; one winner per unit under contention	Retries and failures under heavy write contention; the loser path must be handled
Idempotency keys	Safe retries, no double-reserve or double-charge	A keyed dedup store with its own TTL; the key must scope the operation correctly
Leases	Reserve decoupled from pay; abandoned carts auto-release	A reconciler to reclaim; dependence on clocks; a window where stock is held but unsold
Idempotent confirm	Payment retries are safe; commit applies once	Expired holds must be rejected, forcing the buyer to restart
Eventual reconciliation	Self-heals drift and survives partial failure	Temporary inconsistency; a buggy reconciler amplifies drift instead of fixing it

Scaling it up

A single hot unit defeats optimistic concurrency by thrashing. When thousands of buyers race for one seat or one flash-sale SKU, almost every CAS loses and retries, hammering the row. The production answers are to flatten the spike with a virtual queue that admits buyers at a rate the inventory can absorb (the waiting room ticketing sites use), and for fungible stock to shard the count into N sub-counters — split 1000 units into ten buckets of 100, decrement a random bucket, and let reconciliation rebalance — so contention spreads across rows. Pessimistic SELECT ... FOR UPDATE is the alternative for low-contention correctness-critical paths, trading throughput for simplicity.

Leases across machines need fencing, not just a timer. If a holder pauses (GC, network stall) past its lease expiry, the unit may be reissued to someone else, and the original holder must not be able to confirm afterward. A monotonically increasing fencing token attached to each lease lets the commit reject a stale holder, which is the distributed-lock correctness problem Kleppmann documents; a wall-clock TTL alone is not enough once clocks and pauses enter. The lease store has to be the single source of truth for who holds what.

Idempotency gets harder than a lookup table. Keys need a TTL and correct scoping (per endpoint and arguments, not global), and the genuinely hard case is two retries of the same key racing while the first is still in flight — which requires an insert-or-get that serializes on the key so the second waits for, rather than re-runs, the operation. Carried end-to-end through the payment provider, this is what makes a charge exactly-once over an at-least-once network.

A real checkout spans more than one resource, so the single-row reconciler generalizes to a saga. Reserving inventory, charging payment, and allocating fulfillment touch separate services that cannot share one transaction, so each step has a compensating action — release the hold if payment fails — and events are emitted reliably through a transactional outbox rather than dual-writing. Reconciliation then runs across services, not just within the inventory table, and the chain of holds and sales must remain an auditable ledger because it is money-critical.

Oversell stance is a policy that reshapes the data model. Unique units forbid it absolutely; fungible goods often allow a bounded oversell that backorder or reconciliation later resolves, which changes the conditional write (a soft limit instead of a hard > 0) and the reconciler's job. From here the concrete follow-ons are a virtual-queue / admission-control prototype that rate-limits buyers into a hot inventory, a saga and outbox prototype that coordinates inventory, payment, and fulfillment with compensations, and a fencing-token lease-manager prototype that makes distributed holds safe against paused holders. Each extends this conditional-decrement-plus-lease core without rebuilding it.

References

Kung & Robinson, On Optimistic Methods for Concurrency Control, ACM TODS (1981) — the original CAS-style validation approach: https://dl.acm.org/doi/10.1145/319566.319567
Stripe — Idempotent requests: storing and replaying a response by idempotency key: https://docs.stripe.com/api/idempotent_requests
Kleppmann, How to do distributed locking — fencing tokens and why a lease TTL alone is unsafe: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
Garcia-Molina & Salem, Sagas, ACM SIGMOD (1987) — long-lived transactions with compensating actions: https://dl.acm.org/doi/10.1145/38713.38742
Richardson, Saga and Transactional outbox patterns — cross-service consistency and reliable event emission: https://microservices.io/patterns/data/saga.html · https://microservices.io/patterns/data/transactional-outbox.html
Kleppmann, Designing Data-Intensive Applications — weak isolation, lost update, and write skew: https://dataintensive.net/
Optimistic locking in ORMs (the version-column pattern in production): Hibernate @Version and Rails optimistic locking documentation.
Foundations referenced inline: optimistic concurrency · idempotency · leases · eventual reconciliation