Timeouts and retries

On one machine, a function either returns or throws. Across a network, the caller may stop hearing from a server while that server continues working. Partial failure means some components are healthy, others aren't, and observers can disagree about which is which. Timeouts bound waiting; they don't remove uncertainty.

The unknown outcome

A checkout service sends a charge request. The provider commits the charge, but the response is lost. The checkout service times out. Retrying with a new identity can charge twice; declaring failure can tell the user to try again even though money moved.

Keep the operation pending, look up its status, or retry with the same idempotency key under the provider's contract. The next lesson covers that mechanism. Cancellation is helpful but may arrive after a commit, so it isn't proof that nothing happened either.

Pick timeouts from a budget

Set a total request deadline, then give downstream calls a share of the remaining time. A one-second user request can't sensibly spend one second on each of three sequential dependencies and still meet its goal. Reserve time for local work, returning a response and any permitted retry.

Measure dependency latency under representative load. Choose an acceptable false-timeout rate and inspect the corresponding percentile, then allow for network variation and cold connections. For example, if p99.9 is 80 ms, a 100 ms call budget might be a starting experiment, not a universal safe setting.

Check whether the timeout covers DNS, connection establishment, TLS and reading the response, or just one stage. Use a monotonic clock for elapsed time. Propagate deadlines and cancellation so abandoned work doesn't keep consuming resources unnecessarily.

Back off, with jitter

Retry transient failures only when the operation is safe to repeat. Authentication failures and invalid input won't improve after another attempt. Respect a dependency's Retry-After and don't retry a request whose deadline has already expired.

The delay cap doesn't bound total work: a client can retry forever at the cap. Also limit attempts and total elapsed time. Long-running asynchronous delivery, such as webhooks, needs a retention window, a dead-letter path and visible delivery status.

Retry budgets

Retries add traffic exactly when a dependency is least able to serve it. A retry budget caps extra attempts across requests, perhaps at 10% of original call volume over a window. A token bucket can enforce it; exhausted callers fail or defer rather than creating more work.

At 10,000 original calls a second, a 10% retry allowance permits about 1,000 extra calls a second, not unlimited retries from every caller. During a long outage the budget should shrink or run out rather than masking failure with repeated load.