LESSON 11.4
Quotas, fairness and adaptive limits
Long-term quotas versus short-term rate limits, weighting requests by cost, concurrency limits, fair sharing between tenants, and limits that adapt to how the backend is doing.
6 min read
The algorithms in this module so far answer "how many requests per second?". Real platforms also need longer horizons, requests of different sizes, many tenants sharing capacity, and backends whose capacity changes from minute to minute.
Rate limits and quotas
These are different tools with different jobs:
Rate limit
- Protects capacity right now
- Smooths bursts
- Says nothing about monthly usage
Pick it when: Requests per second or minute, enforced in the request path.
Quota
- Ties usage to a plan or bill
- Long horizon (day, month)
- Counts must be durable and auditable
Pick it when: API plans, storage limits, monthly message allowances.
A quota counter can lag a little behind real time and be slightly over-generous; it must never lose counts, because it feeds billing. That's why quotas are usually counted from a durable usage log, while rate limits live in fast, best-effort counters.
Charge by cost
Requests are not equal. Weighted limits give each request a cost in tokens: a single read costs 1, a search costs 10, a bulk export costs 500. GitHub's GraphQL API does this by computing a cost from the query's shape before running it.
When cost can't be known up front, charge an estimate before and settle the difference after, or put the expensive endpoint behind its own small limit.
Limit concurrency too
Rate limits control how fast requests arrive. Concurrency limits control how many are in progress. They matter when latency changes:
- 100 req/s × 50 ms
- 5 in flight
- 100 req/s × 2 s (backend degraded)
- 200 in flight
- 100 req/s × 30 s (timeouts)
- 3,000 in flight - threads and memory exhausted
A concurrency cap per client (and per endpoint) stops a slowdown from turning into an outage, because it tightens exactly when requests get slower.
Fairness between tenants
In a multi-tenant system, one customer's batch job shouldn't starve everyone else. Common approaches:
- Per-tenant limits sized to their plan, so no tenant can take everything.
- Fair queueing: each tenant gets its own queue and workers take from them in turn (weighted by plan), so a tenant with a million queued jobs doesn't delay one with ten.
- Max-min fair sharing when capacity is short: fully serve small demands, split the rest among large ones.
Adaptive limits
Fixed limits are guesses about capacity. Adaptive limiting measures it: the client or proxy watches latency, and raises its allowed concurrency while latency stays low, then cuts it sharply when latency climbs or errors appear, much like TCP congestion control (additive increase, multiplicative decrease). Netflix's concurrency-limits library and Envoy's adaptive concurrency filter work this way. They don't replace per-client limits, which are about fairness; they protect the backend when its real capacity drops.
Try it
Distributed Quota Accounting HLD
Design quotas shared fairly across tenants and regions.