LESSON 8.2
How load balancers choose
Round robin, least connections and hashing, health checks, and what to do about the load balancer itself being a single point of failure.
6 min read
A load balancer receives every request and passes each one to a server. How it picks the server, and how it notices when a server is sick, decides whether your fleet shares work evenly or some machines drown while others idle.
Balancing algorithms
Round robin
- Simple
- Even request counts
- Ignores how busy each server is
Pick it when: Uniform, short requests on identical servers.
Weighted round robin
- Bigger servers get more traffic
- Weights are static
Pick it when: Mixed instance sizes, or gradually shifting traffic to a new version.
Least outstanding requests
- Adapts to slow requests and slow servers
- Needs per-server tracking
Pick it when: Variable request cost. A strong default for APIs.
Power of two choices
- Pick two servers at random, use the less busy one
- Nearly as good as least-loaded, cheap at scale
- Slightly less even than a global view
Pick it when: Large fleets and distributed load balancers.
Hashing (by user, key or IP)
- The same key always reaches the same server
- Uneven if some keys are hot
- Rebalancing moves keys
Pick it when: When locality matters: caches, sharded in-memory state, connection gateways.
Health checks
Load balancers probe each server, for example GET /health every few seconds. A server failing several checks in a row is taken out of rotation; passing again brings it back.
Two kinds of check are worth distinguishing:
- Liveness: is the process running at all? Failing it means restart the instance.
- Readiness: can it serve traffic right now? A server warming its cache or cut off from its database should fail readiness so it gets no requests, without being killed.
A health check that also checks every dependency can backfire: if the database blips, every server fails its check at once and the load balancer has nobody left to send traffic to. Keep checks focused on the server itself.
Consistent hashing
When requests must go to a particular server, such as the cache node that holds a key, the naive formula hash(key) % N has a nasty property: change N and almost every key maps to a different server. Add one cache node and nearly every lookup misses.
Consistent hashing places servers and keys on a ring. Each key belongs to the next server clockwise. Adding a server only takes over the keys between it and its neighbor, about 1/N of them. Each physical server usually appears at many points on the ring (virtual nodes) so load spreads evenly. Module 15 covers it in depth; for now, know that it's how caches, sharded stores and connection gateways route by key without reshuffling everything on every change.
The load balancer can fail too
A single load balancer is a single point of failure. Common answers:
- Managed load balancers (AWS ALB/NLB, Google Cloud Load Balancing) are themselves distributed across many machines and zones.
- Active-passive pairs share a virtual IP; the standby takes over when the active one fails.
- DNS or anycast spreads users across several load balancers or regions.
In an interview, a single "Load balancer" box is fine, as long as you can say, if asked, that it's a managed, redundant service rather than one machine.
Try it
Distributed Cache HLD
Route keys to cache nodes without reshuffling everything when the cluster changes.