Rate limiting across many servers

One limiter on one server is easy. A real API runs on many gateway servers in several regions, and a client's requests land on all of them. The limit has to hold for the client as a whole, without adding noticeable latency to every request.

Shared counters

The standard design keeps the counter in a fast shared store, usually Redis, keyed by client and route:

Two things make it work:

Redis handles over 100,000 simple operations a second per node. Beyond that, shard counters by key across a Redis cluster; each key lives on one shard, so atomicity still holds.

When a network hop is too much

For extremely high request rates, even a sub-millisecond call per request adds up. Approximations trade precision for speed:

Failure policy

The limiter sits on every request, so its dependencies sit on every request too. Decide in advance what happens when Redis is slow or down:

• Fail open: allow the request, count the failure, alert, and apply a coarse local limit (the global limit divided by the number of servers) so abuse is still bounded. Right for most traffic.
• Fail closed: reject. Right where abuse costs more than an outage: login attempts, payment attempts, expensive operations.
• Always use a short timeout (a few milliseconds) on the limiter call, so a sick Redis can't make every request slow.

Multiple regions

Clients may hit several regions. Options, from simplest:

• Per-region limits: each region enforces its own share. Simple and fast; a client spread across regions gets a bit more.
• Home region: route each client's limit checks to one region. Accurate, but adds cross-region latency for clients far from home.
• Asynchronous global sync: regions enforce locally and exchange counts every second or so. Close to accurate, with a small overshoot.

Most systems choose per-region limits, or local enforcement with async sync, because exact global limits cost too much latency.