Pattern · Consistency
Idempotency
Handling retries safely without duplicate side effects. Idempotency keys, deduplication tables, consumer deduplication, and exactly-once processing via at-least-once delivery with deduplication.
11 min read
In distributed systems, networks drop packets, connections time out, and workers crash after executing work but before acknowledging it. Clients and queues must retry failed requests. Idempotency guarantees that executing an operation multiple times produces the exact same outcome as executing it once, preventing duplicate charges, duplicate orders, and duplicate notifications.
The problem
The classic distributed race occurs on an unacknowledged timeout:
- A client sends
POST /chargesto charge $50. - The payment service charges the credit card and records the transaction.
- The network blips or the server restarts before the HTTP 200 response reaches the client.
- The client experiences an HTTP 504 Gateway Timeout and retries the exact same request.
- Without idempotency, the payment service executes a second charge, billing the customer $100.
True network-level exactly-once delivery is impossible across untrusted networks. Instead, real-world systems achieve effectively once processing by combining at-least-once delivery with idempotent execution.
Idempotency keys
An idempotency key is a unique client-generated token (typically a UUIDv4 or a deterministic business hash like order_id + event_type) sent via the Idempotency-Key HTTP header.
- Client attaches key:
POST /v1/chargesincludesIdempotency-Key: 9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d. - Lock or insert state: The server atomically attempts to insert an idempotency record in state
IN_PROGRESS. - Handle in-flight requests: If an identical key is already
IN_PROGRESS, concurrent duplicate requests return HTTP 409 Conflict or wait on a lease to prevent concurrent double execution. - Execute operation: The server performs the non-idempotent side effect (calls the credit card gateway, transfers funds).
- Persist cached response: The server transitions the idempotency record to
COMPLETEDand saves the HTTP status code and response JSON. - Replay on retry: When the client retries with the same key, the server detects state
COMPLETEDand immediately replays the saved response without re-executing the payment.
Where to store keys and for how long
Relational Database Table
- Atomic transaction with business mutation
- Never loses key if DB commits
- Adds write load to primary database
Pick it when: Critical transactional operations (payments, account balances, orders).
Redis Key-Value Cache
- Sub-millisecond lookup latency
- Native TTL expiration (SET key val EX 86400 NX)
- Cache eviction or failover can lose keys
- Separate system from database
Pick it when: High-volume notifications, webhooks, non-financial API requests.
Key retention windows
Idempotency keys do not need to live forever. Clients typically retry within seconds, minutes, or at most a few days after network restoration.
- Transactional APIs (Stripe, Adyen): Retain keys for 24 to 72 hours. After 72 hours, keys expire, and subsequent requests with the old key are rejected or treated as new.
- Storage size: 10 million daily operations × 100 bytes per idempotency record ≈ 1 GB of storage per day, easily managed with a TTL or a daily time-partitioned table.
Idempotent consumers in message queues
When consuming from message queues (such as SQS, RabbitMQ, or Kafka), redelivery is guaranteed. Workers must be idempotent:
Deduplication table within database transaction
- 100% atomic with business state mutation
- Zero window for duplicates
- Requires relational database on the consumer path
Pick it when: Workers writing to a database: insert message_id into processed_messages in the same transaction as updating user status.
Distributed Lock / Check-and-Set in Redis
- Fast, works across non-database sinks
- Small crash window between processing and acking
Pick it when: Calling external third-party APIs (sending emails, calling external webhooks).
Naturally idempotent operations
- No state storage needed
- Simplest to operate
- Only applies to specific mutations
Pick it when: SET status = 'PAID' or DELETE /users/42 instead of INCREMENT or APPEND.
Concrete examples
1. Payment processing (Stripe pattern)
Store request body hashes alongside the idempotency key. If a client sends the same Idempotency-Key with a different request payload (e.g. changing the amount from $50 to $500), immediately return HTTP 400 Bad Request to prevent payload tampering.
2. Notification delivery
A worker delivers SMS login codes. If the SMS vendor takes 5 seconds to reply, the worker's visibility timeout could expire, causing a second worker to pick up the same job. The worker creates a unique key (notification_id + channel), checks a local sent-log before sending, and forwards the stable key to the SMS vendor if the vendor supports idempotency keys.
Numbers to quote
- Key retention windowcovers client retry windows and network restoration
- 24–72 hours
- Deduplication lookup overhead
- < 2 ms in Redis, 3–5 ms in indexed SQL table
- Idempotency key format
- UUIDv4 (36 chars) or sha256 hash (64 chars)
- Payload hash verification
- sha256 of request body to detect payload tampering
Where it bites
- Lease expiration on slow work: If state is marked
IN_PROGRESSwith a 30-second lease, and execution takes 35 seconds, a retry will acquire the lease and run concurrently. Refresh leases on long-running tasks. - Ignoring idempotency in consumers: Assuming queues deliver exactly once. SQS standard and Kafka consumer rebalances will redeliver messages.
- Mismatched client keys: Clients generating a new random UUID on every retry attempt defeat the entire idempotency system. The retry loop must preserve the original key.
Try it
E-commerce Checkout HLD
Design an e-commerce checkout workflow that guarantees no double-charging on retried requests.