Safe effects and isolation

Two problems often arrive together: a caller repeats an operation after an unknown outcome, and a slow dependency consumes every available worker. Idempotency protects the business effect. Isolation and admission control protect capacity. A healthy design needs both.

Give the operation an identity

For an operation such as "place this order," the client creates an idempotency key and reuses it for retries of the same intent. Scope it to the authenticated customer and operation. Store a fingerprint of the relevant input so reusing the key with a different basket is rejected, not mistaken for a retry.

A cache-only key can disappear on restart or eviction. A check followed by an unrelated mutation can race. Make ownership, mutation and outcome recording atomic at the boundary you control. If processing is long-running, a durable state machine replaces one huge transaction.

External effects need their own boundary

A database transaction can't atomically commit a third-party charge. Record the intent, call the provider with a stable derived key, and reconcile uncertain results using that same identity. Don't generate a new provider key on each worker attempt.

An outbox can publish local committed changes reliably, but its consumer still needs dedupe. With at-least-once delivery, a worker may finish and crash before acknowledging. Record the message id together with the local effect, then acknowledge after commit. For an external effect, use the downstream idempotency contract or explicitly handle possible duplicates.

Circuit breakers

A breaker watches recent dependency failures and switches between three states:

• Closed: calls proceed, with bounded timeouts and failure measurement.
• Open: calls fail quickly or use a defined fallback for a cooldown period.
• Half-open: a few probes test recovery; successful probes restore traffic, failures reopen it.

Breakers reduce wasted calls but don't cancel work already running. Combine them with concurrency limits. Use minimum sample counts and sensible thresholds so one isolated error doesn't cut off a healthy dependency; recovery probes need jitter too.

Bulkheads and load shedding

A bulkhead reserves resources for separate workloads. Give payment calls and recommendations different concurrency pools. Otherwise slow recommendations can occupy every request worker and prevent checkout even when payments are healthy.

Load shedding rejects work the service can't finish usefully. Reject early with a clear overload response, keep queues bounded, and protect urgent work before optional work. A long queue doesn't add throughput; it can turn overload into timeouts followed by even more retries.