LESSON 10.2
Delivery guarantees and idempotent consumers
At-most-once, at-least-once and the myth of exactly-once, visibility timeouts, and how to make processing a message twice harmless.
6 min read
Queues promise that messages get delivered. The fine print matters: delivered how many times? Most bugs in asynchronous systems come from assuming "exactly once" when the system actually provides "at least once".
Three guarantees
At most once
- Never processed twice
- Messages can be lost
Pick it when: Metrics and logs where a small loss is fine.
At least once
- Never lost
- Sometimes processed twice
Pick it when: The default for almost everything. Pair it with idempotent consumers.
Exactly once
- What everyone wants
- Only achievable within one system's boundaries, with care
Pick it when: Kafka transactions inside Kafka; otherwise build it as at-least-once plus idempotency.
The order of two steps decides which you get:
- Process, then acknowledge. A crash between them means the message is redelivered and processed again: at least once.
- Acknowledge, then process. A crash between them means the message is gone: at most once.
Since losing work is usually worse than repeating it, systems process then acknowledge, and design consumers so repeats don't matter.
Visibility timeouts and leases
When a worker receives a message from a queue like SQS, the message isn't deleted; it becomes invisible for a timeout, say 30 seconds. If the worker acknowledges in time, the message is deleted. If the worker crashes or takes too long, the message reappears for another worker.
That's how queues survive crashed workers, and also how duplicates happen: a slow worker that exceeds the timeout may still finish while a second worker processes the same message. Set the timeout above the normal processing time, and extend it for long jobs.
Idempotent consumers
An operation is idempotent if doing it twice has the same effect as doing it once. Make every consumer idempotent:
- Natural idempotency. "Set status to shipped" is safe to repeat; "add 1 to the counter" is not. Prefer absolute writes over increments where you can.
- Deduplicate by message id. Record processed ids in a table with a unique constraint, in the same transaction as the work. A repeat hits the constraint and is skipped.
- Pass idempotency keys downstream. When calling a payment provider or another API, send a key derived from the message so they deduplicate too.
handle(message):
begin transaction
insert into processed_messages (id) values (message.id) -- unique; fails on repeat
apply the change
commit
ack(message)Retries and dead letters
Failures come in two kinds. Transient ones (a timeout, a 503) succeed if you try again later. Permanent ones (malformed data, a deleted account) never will.
- Retry transient failures with exponential backoff and jitter: 1 s, 2 s, 4 s, 8 s plus randomness, so retries don't arrive in synchronized waves.
- Cap the attempts. After the limit, move the message to a dead-letter queue (DLQ) and alert. Someone can inspect, fix and replay it.
Try it
Webhook Delivery HLD
Deliver events to customers at least once, with retries, backoff and a dead-letter queue.