Technology · Coordination service
ZooKeeper and etcd
Small, strongly consistent stores for coordination. What they guarantee, leases and watches, leader election, service discovery and configuration, and what not to put in them.
10 min read
ZooKeeper and etcd are replicated, strongly consistent key-value stores built on consensus (ZAB for ZooKeeper, Raft for etcd). They hold a small amount of critical data that many machines must agree on: who the leader is, which nodes are alive, what the current configuration is. Kubernetes stores all its state in etcd; older Kafka, HBase and Hadoop used ZooKeeper.
When to reach for them
- Leader election: exactly one scheduler, controller or primary at a time.
- Locks and leases that must be correct, not just usually correct.
- Service discovery and membership: which instances are alive, and where.
- Configuration and feature flags that must change everywhere consistently.
- Shard assignment: which node owns which partition.
What they guarantee
- A cluster of 3 or 5 nodes elects a leader. Writes go through the leader and commit once a majority has them.
- Writes are linearizable: once acknowledged, every later read that asks for it sees the write.
- Every change gets a monotonically increasing revision (etcd) or zxid (ZooKeeper), usable as a fencing token.
- A minority partition can't accept writes, so the cluster never has split brain. With 5 nodes it tolerates 2 failures.
Building blocks
Leases / ephemeral nodes
- Vanish when the client's session or lease expires
- A paused client may not know it lost the lease
Pick it when: Liveness, membership, locks.
Watches
- Clients are notified of changes instead of polling
- Can miss intermediate values; re-read on notification
Pick it when: Config changes, leader changes, membership.
Compare-and-swap / transactions
- Atomic conditional updates
- Small payloads only
Pick it when: Elections, claiming a shard, updating config safely.
Sequential nodes (ZooKeeper)
- Ordered queue of contenders
- ZooKeeper-specific
Pick it when: Fair locks and elections without thundering herds.
Leader election in practice
- Each candidate tries to create the key
/service/leaderwith its id, attached to its lease, only if the key doesn't exist. - The winner keeps renewing its lease and acts as leader, including the key's revision as a fencing token in its writes.
- The others watch the key. When the leader's lease expires, the key disappears and they try again.
Numbers and limits
- Cluster sizemore nodes make writes slower
- 3 or 5 nodes
- Writesevery write is a consensus round
- thousands to ~10,000+ per second
- Data sizeetcd defaults to a 2 GB quota
- a few GB at most
- Value sizeetcd limits requests to ~1.5 MB
- KB, not MB
Where they bite
- Sessions and GC pauses: a client paused past its lease timeout loses leadership without noticing. Always fence writes with the revision.
- Cross-region clusters pay a WAN round trip on every write; usually run one per region.
- Watch storms: thousands of clients re-reading after every change can overload the cluster.
Try it
Distributed Lock Service HLD
Design a lock service that stays correct through pauses and partitions.