Pattern · Traffic
Backpressure and Load Shedding
Protecting overloaded services from catastrophic cascading failure. Bounded queues, failing fast at the perimeter, priority-based shedding, circuit breakers, retry storms, retry budgets, and adaptive concurrency limits.
11 min read
When a service receives more requests than its compute, memory, or database capacity can sustain, latency increases. As latency increases, callers hold connections open longer, connection pools fill, thread pools saturate, and memory usage climbs until the process crashes from an Out-of-Memory panic. Surrounding microservices then cascade into failure as they route traffic to remaining overwhelmed nodes.
Backpressure and load shedding are defensive architectural patterns that prevent this death spiral. Instead of attempting to process every request and collapsing, an overloaded service deliberately drops or rejects lower-priority traffic immediately to keep good throughput flowing for critical operations.
The death spiral of unbounded queues
The most destructive mistake in systems engineering is placing an unbounded in-memory queue in front of a bottlenecked resource:
- A database slows down, increasing API request latency from 20 ms to 2,000 ms.
- Inbound requests arrive at 1,000 requests/sec, but the database only completes 100 requests/sec.
- The server buffers the excess 900 requests/sec in an in-memory queue.
- By the time a request is finally dequeued after sitting in line for 25 seconds, the client's HTTP gateway has already timed out (at 5 seconds) and disconnected.
- The server burns expensive CPU and database cycles computing an answer that no client is waiting for.
- The queue consumes all heap memory, garbage collection pauses freeze the runtime, and the node crashes.
Rejecting early: Little's Law and bounded queues
Every buffer in a production system must be strictly bounded.
Under Little's Law, the average number of concurrent requests in a stable system ($L$) equals the arrival rate ($\lambda$) multiplied by the average response time ($W$):
Concurrent Requests (L) = Inbound Rate (lambda) * Latency (W)If a service can handle at most 200 concurrent requests before CPU saturation or database connection pool exhaustion, any request arriving when 200 requests are already in-flight should not wait in a queue for minutes. It should be rejected immediately with an HTTP 429 Too Many Requests or HTTP 503 Service Unavailable:
Priority-based load shedding
Not all traffic is created equal. When a service detects that latency is degrading or CPU exceeds 85%, it should shed load based on request criticality:
- Critical path (P0): Checkout transactions, core messaging, payment processing, authentication. Never shed unless hardware is at immediate risk of collapse.
- Standard user interactions (P1): Profile updates, comment posts, search queries. Served if capacity permits; throttled under heavy load.
- Background and telemetry (P2): Analytics beacons, click tracking, audit ingestion, asynchronous thumbnail generation. Shed aggressively during peak surges.
- Internal crawlers and batch jobs (P3): Search indexing crawlers, data warehouse exports. Completely suspended until operational health stabilizes.
Ingress proxies inspect an HTTP header (such as X-Priority: high or authentication claims) and reject low-priority traffic at the perimeter before it touches backend microservices.
Circuit breakers: Failing fast on downstream distress
When an upstream service calls a downstream dependency that is failing or timing out, waiting for long timeouts wastes threads:
A circuit breaker operates as a state machine:
- Closed: Requests pass through normally. Success and failure rates are tracked.
- Open: If failure rates exceed a threshold (e.g. 50% errors over 10 seconds), the breaker trips open. All subsequent calls immediately fail fast with an error or return a stale cached fallback without hitting the network.
- Half-Open: After a cooldown period (e.g. 30 seconds), a single probe request is allowed through. If it succeeds, the breaker closes and resumes normal traffic; if it fails, the cooldown timer resets.
Stopping retry storms with retry budgets
When a service stutters, naive clients immediately retry failed requests. If 1,000 clients each retry 3 times, an already struggling service receives $3\times$ more load, cementing an outage.
To eliminate retry storms:
- Exponential backoff with full jitter: Retries wait exponentially increasing random intervals (
wait = random(0, base * 2^attempt)), desynchronizing thundering herds. - Retry budgets: A client or gateway tracks the ratio of retries to initial requests. If retries exceed 10% of total outbound requests, all further retries are suppressed.
- Never retry client errors: Never retry HTTP 4xx errors (bad input, unauthorized); only retry transient 503 or network socket timeouts.
Adaptive concurrency limits
Static concurrency limits (e.g. "max 100 concurrent requests") are brittle: a database might handle 500 simple reads but choke on 40 complex queries.
Modern systems use adaptive concurrency limits (modeled after TCP congestion control algorithms like TCP Vegas or Netflix's Gradient limit):
- Measure baseline P90 round-trip time (RTT) under low load (
rtt_min). - Continuously observe current round-trip time (
rtt_current). - If
rtt_currentbegins climbing abovertt_min * 1.5, queues are building up inside dependencies. - Dynamically contract the maximum allowed concurrent requests down until latency recovers, shedding excess requests at the ingress gate.
Comparing defensive overload mechanisms
Bounded In-Memory Queue
- Absorbs micro-bursts of a few hundred milliseconds without dropping requests
- Unbounded queues guarantee Out-of-Memory crashes
Pick it when: Short-lived spikes within known memory headroom.
Priority Load Shedding
- Protects revenue-generating core flows during extreme 10x traffic spikes
- Requires careful classification and header propagation across services
Pick it when: Large microservice architectures, e-commerce checkout systems.
Circuit Breakers
- Prevents a single failing dependency from hanging all upstream threads
- Requires defining meaningful fallback responses
Pick it when: External payment gateways, third-party APIs, non-critical sub-queries.
Adaptive Concurrency Limits
- Zero manual threshold tuning; auto-adapts to variable database capacity
- Requires fine-grained telemetry and math in the RPC client layer
Pick it when: High-throughput RPC communication between internal services.
Numbers to quote
- Max queue wait duration
- Should never exceed client timeout (typically 200 ms to 1 s)
- Retry budget limit
- Cap retries at 10% to 20% of total outbound traffic
- Circuit breaker error trip threshold
- 50% failure rate over a rolling 10-second window
- Circuit breaker half-open cooldown
- 15 to 30 seconds before probing recovery
- Rejection response code
- HTTP 429 (rate limit) or HTTP 503 (temporary capacity limit)
Where it bites
- Dropping the wrong requests: If priority headers are missing or spoofed by external users, high-value checkouts may be shed while low-priority telemetry passes through. Always sanitize and re-classify priority headers at the API gateway perimeter.
- Deadlocks in half-open state: If the single probe request sent during half-open hits a cold database connection and times out, the circuit breaker remains locked open unnecessarily. Use small health-check pings or route a modest percentage (1% to 2%) of real traffic.
Try it
Distributed API Rate Limiter HLD
Design an edge rate-limiting and load-shedding tier that shields internal microservices during sudden traffic surges.