Email Service
Send, receive, and store mail reliably: a durable queue, exponential backoff retries, a dead-letter path, idempotent accept and delivery, and hot/cold storage tiering.
~30 min · intermediate
Problem & Requirements
Accept outbound mail, deliver it to remote servers, accept inbound mail, and store all of it durably. The defining constraint is that delivery is unreliable and slow by nature: the recipient's server may be down, rate-limiting, greylisting a first attempt, or rejecting the message outright, and the only correct response to "temporarily unavailable" is to try again later — sometimes for days. An email system is mostly a machine for retrying delivery without losing messages, sending duplicates, or retrying forever.
That shape rules out doing the work inside the request that submitted the mail. Accepting a message and delivering it have to be decoupled through a queue so the caller gets a fast, durable acknowledgment while delivery happens asynchronously. Transient failures demand retries with backoff so the system doesn't hammer a struggling server. Permanent failures and exhausted retries need a dead-letter path so nothing is silently dropped or retried indefinitely. Because the queue is at-least-once and clients themselves retry, every step has to be idempotent or the same message goes out twice. And the stored corpus grows without bound, so message bodies move across storage tiers as they age. Those five principles are the prototype.
Functional
accept(message, idempotency_key)durably takes outbound mail and returns fast; delivery happens later.- Deliver to the recipient's mail server, retrying transient failures and dead-lettering permanent ones.
- Store every message durably and retrievable by id, with recent mail fast to read.
Non-functional (back-of-envelope, large mail system)
| Quantity | Target | What it forces |
|---|---|---|
| Send volume | 10⁶–10⁹ messages/day | Accept and delivery must be separate; delivery is a worker fleet, not a request |
| Accept latency | < ~100 ms | Accept only enqueues + persists; it never connects to a remote server |
| Delivery window | seconds to ~4–5 days | Long-running retries with backoff, bounded by total age (standard SMTP practice) |
| Transient failure rate | high and normal | Greylisting/rate-limits are expected; backoff + jitter, not immediate re-send |
| Delivery semantics | at-least-once queue | Redelivery is guaranteed to happen; delivery must dedupe to stay at-most-once visible |
| Body size / corpus | KB–MB each, unbounded total | Bodies separate from metadata; recent in a hot store, old in cheap cold storage |
The semantics row is the one that quietly causes duplicate-email bugs. A durable queue redelivers a job whenever a worker crashes after sending but before acknowledging, and clients re-submit when an accept times out, so duplicates arrive at two distinct points. Idempotency has to be enforced at both — accept and delivery — which is step five.
Design
Six components, each tied to its principle:
- Accept API — validates the message, resolves or assigns an idempotency key, writes the body to the store, and enqueues a delivery job. It returns once the job is durably queued, not once the mail is delivered.
- Durable queue — holds delivery jobs and supports delayed re-enqueue for backoff. The decoupling layer; see queues. It is at-least-once, which the design treats as a guarantee to defend against, not a flaw.
- Delivery workers — dequeue a job, look up the recipient's mail exchanger, attempt SMTP delivery, and classify the outcome as success, transient failure, or permanent failure.
- Retry scheduler — on a transient failure, re-enqueues the job with an exponentially growing, jittered delay, bounded by a maximum attempt count and total age. This is retries with backoff.
- Dead-letter queue — receives jobs that hit a permanent failure or exhaust their retry budget, so they can be bounced back to the sender or inspected rather than lost or looped forever. See dead-letter queues.
- Tiered store — keeps small message metadata indexed and separate from large bodies, with recent bodies in a fast hot store and aged bodies migrated to cheap cold object storage. This is storage tiering, and the inbound (receive) path lands here too: an MX server accepts inbound SMTP, queues the message for spam and virus processing, then writes it to the same tiered store.
Two real systems anchor the halves. Amazon SES is the managed sending side made concrete: it accepts mail, retries delivery, and exposes the failure outcomes as bounce and complaint notifications plus a suppression list, which is a productized version of the dead-letter and idempotency machinery here. The Gmail backend anchors receive-and-store: ingesting and indexing mail at scale, with message data spread across Google's storage stack and aged across tiers. SES shows where the queue/retry/DLQ effort goes; Gmail shows where the storage-tiering effort goes.
Build it
The naive version delivers inside the call: look up the recipient's mail server and hand off the message over SMTP, right there. It works when the remote server is healthy and reachable, and it fails every other time — a slow or down recipient blocks the caller, a transient rejection loses the message, and a crash mid-send leaves no record of what happened. Every later step exists to remove one of those failure modes, starting with the coupling itself.
Split accept from deliver. Accept persists the message body, enqueues a delivery job, and returns — so the caller waits on a local write, never on a remote server. A separate worker pulls jobs and delivers them on its own time. The queue is the seam that makes accept fast and delivery independently scalable, and because the job is durably enqueued before accept returns, a crash after acknowledgment doesn't lose the message. The in-memory deque here stands in for a persistent queue.
A transient failure — connection refused, a 4xx greylist, a rate-limit — means "try again later," and trying again immediately just adds load to a server that's already struggling. Re-enqueue the job with a delay that grows exponentially per attempt and add random jitter so a fleet of workers doesn't synchronize into a thundering retry. The delay is capped so it doesn't grow absurdly. This is retries with backoff; the delayed queue is what holds a job until its next attempt is due.
Backoff handles "try again," but two cases must stop the loop: a permanent failure (a 5xx rejection, an invalid address) where retrying is pointless, and a transient failure that has exhausted its budget — too many attempts or too old. Without a bound, a bad address retries forever and clogs the workers. Route those jobs to a dead-letter queue, where they can be bounced back to the sender or inspected, instead of being dropped or looped. The DLQ is what turns "lost mail" into "visible failure."
The queue is at-least-once and clients retry on timeout, so the same message arrives twice at two points. Defend both. At accept, dedupe on a client-supplied idempotency key: a repeated key returns the original message id and enqueues nothing new, so a client retry can't create a second email. At delivery, record a message as delivered the moment SMTP succeeds and skip any redelivered job for an already-sent message, so a worker that crashed after sending but before acknowledging doesn't send a duplicate. This is idempotency applied at the two places duplicates enter.
The stored mail grows without bound, and keeping everything in a fast store is wasteful since old messages are read rarely. Separate small metadata (sender, recipient, timestamps, status) — which stays indexed and queryable — from large bodies, and keep bodies in a fast hot store only while they're recent. A background pass migrates aged bodies to cheap cold object storage, and reads fall back to cold on a hot miss. This is storage tiering: hot for the working set, cold for the long tail, metadata always close at hand.
Tradeoffs
| Decision | What it buys | What it costs |
|---|---|---|
| Queue between accept and deliver | Fast durable accept; delivery scales independently | Eventual delivery, not immediate; a queue to operate and persist |
| At-least-once delivery | Simple, crash-safe redelivery | Duplicates unless every step is idempotent |
| Backoff + jitter | Doesn't overload struggling recipients; avoids synchronized retries | Delivery can take hours; a tuned backoff curve and cap |
| Dead-letter queue | Failures are visible and bounded, never silently dropped or looped | A second queue and a bounce/inspection process to run |
| Idempotency at accept + deliver | At-most-once visible sends despite retries everywhere | An idempotency store with TTLs; dedupe state to maintain |
| Storage tiering | Cheap storage for the long tail; fast access to recent mail | Cold reads are slow; a migration job and per-message tier tracking |
Scaling it up
The toy leaves out most of what makes a real mail system hard. The notable gaps:
The queue and dedupe state are distributed and persistent. An in-process deque and a Python set stand in for a durable, partitioned queue (Kafka, SQS) and a shared idempotency store (a database or a system like the distributed cache from an earlier prototype, with TTLs). The "record delivered before ack" step needs that store to be consistent enough that two workers can't both decide a message is undelivered, which is a real coordination problem the toy waves away.
Deliverability is its own discipline. Getting mail accepted, rather than merely sent, requires SPF, DKIM, and DMARC signing, careful IP-reputation and warm-up management, per-recipient-domain rate shaping, and handling bounces and spam complaints as first-class signals that feed a suppression list. SES exposes exactly these as managed features because they're the hard part of sending at scale, not the SMTP call itself.
The receive path is a pipeline. Inbound mail arrives at MX servers that must accept SMTP under load, then run spam and virus filtering, threading, and indexing before storing — and each stage is queued and retried much like the send path. The store then has to support full-text search over the corpus, which is a separate indexing system on top of the tiered storage.
Tiering is multi-level and dedup-aware. Production storage isn't two tiers but several, with replication and erasure coding for durability, and large mail systems deduplicate identical attachments and message bodies across recipients so a message sent to a thousand inboxes is stored once. Migration runs continuously against access patterns, not on a single age threshold.
Backoff per recipient domain, not per message. A single global backoff is crude; real senders track health per destination domain and back off all mail to a domain that's rate-limiting, while continuing to deliver elsewhere. The retry decision is informed by aggregate signals, not just one job's attempt count.
This is the first messaging-and-reliability prototype, and its pieces recur across distributed systems generally. The natural next steps are a dedicated durable queue prototype (the partitioned, persistent, at-least-once log this lesson assumes), an idempotency / exactly-once-effects prototype that builds the dedupe store and its consistency guarantees properly, and a webhook or notification delivery prototype, which is the same accept-queue-retry-DLQ pattern applied to outbound HTTP rather than SMTP. Each reuses this foundation set directly.
References
- Amazon SES — managed sending with retries, bounce/complaint handling, and suppression lists: https://docs.aws.amazon.com/ses/latest/dg/Welcome.html
- RFC 5321 (SMTP) — the protocol and its retry/queue semantics, including the multi-day retry expectation: https://datatracker.ietf.org/doc/html/rfc5321
- AWS SQS dead-letter queues — the productized DLQ pattern and redrive: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- Stripe on idempotency keys — accept-side dedupe done right, directly applicable here: https://docs.stripe.com/api/idempotent_requests
- Foundations referenced inline: queues · retries with backoff · dead-letter queues · idempotency · storage tiering