Technology · Traffic routing
Load Balancers
Distributing network traffic across backend server pools. Layer 4 vs Layer 7 routing, balancing algorithms, health checks, connection draining, and load balancer redundancy.
11 min read
A load balancer sits between clients and a backend server pool, distributing incoming network traffic so no single server saturates, fails, or degrades application response times. Beyond spreading requests, modern load balancers provide health probing, TLS termination, protocol negotiation (HTTP/2, HTTP/3), and graceful zero-downtime rolling deployments.
When to reach for one
- Horizontal server scaling: Spreading traffic across dozens or hundreds of stateless application instances.
- High availability and failover: Detecting failed backend instances via health checks and routing traffic around them in seconds.
- TLS offload: Terminating expensive TLS handshakes at the edge, freeing backend CPU cores for application business logic.
- Content-based routing: Routing requests to specialized backend microservices based on URL path (
/api/billingvs/api/search), HTTP headers, or hostnames.
Layer 4 vs Layer 7 load balancing
The primary technical distinction is the OSI layer at which routing decisions occur:
Layer 4 (Transport / TCP / UDP)
- Ultra-high throughput: millions of packets per second per instance
- Low CPU overhead: packets routed without inspecting application payload
- Protocol agnostic: handles databases, WebSockets, custom TCP
- No routing by URL path, headers, or cookies
- No HTTP-level caching or compression
- Cannot inspect payloads for WAF rules
Pick it when: Edge network entry points (AWS NLB, Maglev, IPVS, HAProxy TCP mode).
Layer 7 (Application / HTTP / gRPC)
- Rich content routing: path, headers, query parameters, cookies
- TLS termination and certificate management
- Smart connection pooling and HTTP/2 multiplexing
- Higher CPU memory consumption
- Must buffer and parse complete HTTP headers before routing
Pick it when: Web applications, microservices, gRPC backends (AWS ALB, Nginx, Envoy).
Load balancing algorithms
How the balancer chooses which healthy backend receives the next request:
Round Robin & Weighted Round Robin
- Simple, uniform distribution
- Weights handle servers with unequal CPU/RAM capacity
- Ignores actual server load or request duration variations
Pick it when: Homogeneous requests with uniform execution times.
Least Connections & Least Request
- Routes to the server currently handling the fewest active connections
- Prevents slow queries from piling up on one server
- Slightly higher tracking overhead
Pick it when: Long-running requests, database connection pools, file uploads.
Consistent Hashing (IP / Header)
- Hashes client IP or session token to the same backend server
- Minimizes key movement on pool changes
- Can cause traffic skew if one client sends disproportionate traffic
Pick it when: In-memory caching tiers, stateful WebSockets, local affinity.
Peak EWMA (Exponentially Weighted Moving Average)
- Measures latency history and active requests
- Steers away from struggling servers instantly
- Complex algorithmic tuning
Pick it when: High-performance microservice meshes (Envoy, Finagle).
Health checks and failover
A load balancer must never send requests to a dead or struggling instance:
- Active probes: The load balancer sends periodic requests (e.g.
GET /healthzevery 5 seconds). - Threshold detection:
- Healthy threshold: 2 consecutive 200 OK responses mark a node healthy.
- Unhealthy threshold: 3 consecutive failures (timeouts or 5xx responses) mark a node dead.
- Traffic removal: The instant a node fails the threshold, the balancer removes it from the routing table. Zero new connections are routed to it.
- Passive health checks (Outlier detection): In advanced balancers (like Envoy), if an instance returns five consecutive 503 errors on live user traffic, it is automatically ejected for 30 seconds without waiting for the next scheduled probe.
Connection draining (Graceful shutdown)
When deploying a new version or terminating an autoscaling cloud VM, abruptly killing instances severs in-flight payments and file uploads:
- When an instance is marked for deregistration, the balancer stops forwarding new requests to it.
- Existing in-flight connections are allowed a deregistration delay / connection draining window (typically 30–60 seconds) to complete cleanly.
- Once active connections reach zero or the timeout expires, the instance is safely terminated.
Sticky sessions vs stateless backends
Sticky sessions (session affinity) use an HTTP cookie to force all requests from a specific user to hit the exact same backend instance.
- Why teams reach for it: Storing user session state or shopping carts in local web server RAM.
- Why it breaks: If server 4 dies, all its pinned users lose their carts. Furthermore, a single viral crawler or corporate office sharing a single NAT gateway IP concentrates massive traffic onto one server.
- The better pattern: Make web servers completely stateless. Store user sessions in a shared Redis cluster, allowing any server to handle any request.
Load balancer high availability and redundancy
If the load balancer itself is a single machine, it is a single point of failure:
- BGP Anycast (Tier 1 Global Scale): Multiple load balancer instances in different racks or regions advertise the identical public IP address via BGP. Routers distribute traffic across the balancers at the network switch level using Equal-Cost Multi-Path (ECMP).
- DNS Round Robin: Pointing a domain name to multiple distinct load balancer public IP addresses in DNS records.
- Active-Passive VRRP / Keepalived: Two load balancer nodes share a virtual floating IP address. If the active master misses heartbeats, the standby node claims the IP via ARP within 2 seconds.
Numbers to quote
- Layer 4 NLB throughput
- Millions of packets/sec per node, gigabits to terabits/sec
- Layer 7 ALB / Nginx throughput
- 20,000 to 80,000 requests/sec per multi-core node
- Health check interval
- 5–10 seconds typical, with 2–3 failure thresholds
- Connection draining timeout
- 30–60 seconds for web apps; 5–15 minutes for long uploads
- TLS termination latency overhead
- 1–2 ms on modern hardware with session tickets (TLS 1.3)
Where it bites
- Hot spot imbalance behind sticky sessions: Pinned sessions prevent load balancers from redistributing traffic when new server capacity autoscales up.
- Idle connection timeouts on WebSocket and SSE streams: Cloud load balancers terminate idle TCP connections after 60 seconds by default. Long-lived streaming connections must send periodic heartbeat pings (e.g. every 25 seconds).
- Client IP masking: Because the load balancer terminates the TCP connection, backend servers see the load balancer's IP address rather than the client's. Balancers must inject
X-Forwarded-ForandX-Forwarded-Protoheaders, and application rate limiters must inspect those headers.
Try it
API Gateway HLD
Design an ingress layer with load balancing, health checks and traffic routing.