Pattern · Coordination
Leader Election
Ensuring exactly one coordinator manages distributed state without split-brain anomalies. Why one leader is needed, time-bounded leases, fencing tokens for stale leaders, ZooKeeper and etcd consensus recipes, and failover timing trade-offs.
10 min read
In distributed architectures, many operations must not run concurrently across multiple machines: assigning partition ownership, scheduling cron jobs, running cluster rebalancing, or closing an auction. Leader election is the pattern where a pool of identical worker nodes dynamically coordinates to select exactly one active leader to execute centralized decisions, while standby followers remain ready to take over if the leader fails.
Why have a single leader
While fully decentralized peer-to-peer architectures eliminate single bottlenecks, many system operations require global serialization:
- Serial decision making: A cron scheduler (like
job-scheduler) must never launch two instances of the same billing reconciliation batch job. - Shard and partition assignment: Orchestrating which Kafka broker or database replica owns which data partition requires an authoritative control plane.
- Preventing resource contention: In distributed lock managers, a designated leader sequences acquire requests without costly peer gossip rounds.
The danger of split-brain
The fundamental challenge in leader election is that networks are unreliable and processes experience unpredictable pauses (garbage collection pauses, thread starvation, VM migrations).
To prevent split-brain, leader election relies on leases and fencing tokens.
Leases: Time-bounded leadership
A leader does not own leadership permanently; it acquires a lease valid for a finite duration (e.g. 10 seconds).
- Active renewal: The leader runs a background heartbeat thread that repeatedly renews its lease before expiration (e.g. every 3 seconds for a 10-second lease).
- Voluntary abdication on renewal failure: If network connectivity drops or the node pauses long enough that the lease cannot be renewed, the leader must immediately revoke its own privileges and stop executing mutations.
- Standby takeover: Standby candidates watch the lease. Only when the full 10-second lease expires without renewal do followers initiate a new election round.
Fencing tokens: Defending against stale leaders
Because a paused node cannot know that its lease expired while it was asleep, downstream storage systems must reject commands issued by demoted leaders. This is enforced with fencing tokens:
- Every time a new lease is granted, the consensus store increments a monotonically increasing counter (e.g. token 34 for Node A, token 35 for Node B).
- The leader includes its current fencing token with every mutation sent to downstream services or databases.
- The storage engine tracks the highest fencing token it has ever observed. If Node A wakes up and sends a write with token 34 after the database has already processed token 35 from Node B, the database immediately rejects Node A's write with an error.
Election recipes in etcd and ZooKeeper
Building leader election directly on raw relational databases or basic Redis keys is error-prone. Production systems rely on consensus stores like ZooKeeper and etcd:
- etcd Concurrency API: Candidates attempt to create a key (
/services/scheduler/leader) attached to an etcd lease with a short TTL (e.g. 5 seconds) using a Compare-And-Swap transaction (createRevision == 0). The winner becomes leader. The leader continuously sends KeepAlive heartbeats. Losers set an event watch on the leader key. When the key is deleted upon lease expiration, all watchers wake up and attempt to acquire. - ZooKeeper Ephemeral Sequential Nodes: Candidates create an ephemeral sequential znode under
/election/leader-. ZooKeeper assigns increasing sequence numbers (leader-001,leader-002). The candidate with the lowest sequence number becomes leader. Each candidate watches only the znode immediately preceding its own number, avoiding the "thundering herd" problem when the leader dies.
Tuning failover timing: Availability vs safety
Choosing lease TTLs is a strict trade-off:
Aggressive Failover (1–2s TTL)
- Fast recovery during genuine crashes (under 3s downtime)
- Minor network hiccups or brief GC pauses trigger false failovers and unnecessary leader flapping
Pick it when: Stateless coordinators with trivial handover overhead.
Conservative Failover (10–30s TTL)
- High stability; tolerates temporary network spikes and 5s GC pauses without flapping
- Cluster waits the full TTL duration before electing a successor during hard server crashes
Pick it when: Heavy stateful controllers (HDFS NameNode, database primaries).
Numbers to quote
- Recommended lease TTL
- 5 to 15 seconds for cloud microservices
- Heartbeat interval
- TTL / 3 (e.g. send heartbeat every 3s for a 10s lease)
- Failover detection delay
- Lease TTL + election round-trip (typically 6 to 18 seconds)
- Clock drift margin
- Never allow clock skew between nodes to exceed 50% of the lease TTL
- Consensus quorum size
- 3 nodes tolerates 1 failure; 5 nodes tolerates 2 failures
Where it bites
- The thundering herd: If 500 worker pods all wake up simultaneously on a single lock key deletion, they flood the consensus cluster with concurrent write transactions. Use ZooKeeper sequential nodes or randomized jitter before re-attempting acquisitions in etcd.
- Async storage writes: If the leader writes to asynchronous queues without waiting for confirmation, a stale leader can queue duplicate messages before being fenced at the database level.
- Heartbeat thread starvation: If application heartbeats run on the main worker thread pool instead of an isolated high-priority daemon thread, a heavy CPU spike can delay heartbeats and cause the node to lose its own lease unexpectedly.
Try it
Distributed Job Scheduler HLD
Design a resilient job scheduler that coordinates cron tasks using an active-passive leader election architecture.