Leader election and consensus

A scheduler needs one authority to assign jobs. Choosing the machine with the smallest id works until machines disagree about who's alive. Consensus lets a group agree on an ordered history despite crashes and communication failures. Leader election is one part of that protocol, not a replacement for it.

What agreement means

Imagine three controllers with different ideas about the current owner of a partition. Clients shouldn't get conflicting authoritative assignments just because they reach different controllers. A replicated state machine applies the same committed commands in the same order, producing the same ownership map.

The safety goal is not committing conflicting histories. Liveness is eventually making progress. During prolonged disconnection or unstable timing, a consensus group may preserve safety while failing to make progress. Ordinary Raft assumes crash failures and protocol-following participants, not malicious voters forging messages.

Raft at a high level

Raft divides leadership into increasing terms. A candidate wins a majority of votes under rules that consider log freshness; voters persist their term and vote. The leader appends commands and replicates them to followers.

For an entry from its current term, the leader can establish commitment once it is durably replicated to a majority. Earlier entries become committed through the protocol's ordering rules. Merely finding an old-term entry on several disks isn't the full commit rule. The Raft paper explains why election restrictions and log rules must work together.

Majority and failure domains

These assume the remaining voters can communicate and have suitable durable state. Placement matters: three voters on one host don't survive losing that host. More voters add replication and operational work; they don't automatically increase write throughput.

An isolated old leader may still believe it leads. It can't commit new entries without a majority. Clients may need to retry discovery, and commands retried after a lost response still need stable identities because consensus doesn't deduplicate arbitrary client intent automatically.

Use an established coordination service

etcd provides a Raft-backed key-value API, transactions, leases and revisioned watches. ZooKeeper offers a coordinated namespace and session-based primitives commonly used for election. Their API guarantees differ; don't assume every read, watch or session primitive has identical freshness semantics.

Use established clients and check the required read mode. A stale local read from a replica isn't automatically linearizable because the store uses consensus internally. Membership changes, durable logging, snapshots, election timing and recovery are difficult protocol work; an interview application should usually use a coordination service rather than implement them.

Consensus ends at a boundary

Electing a scheduler leader doesn't revoke the old process's ability to call a payment API or write a file. Fence its downstream writes, or make the effect safe through an authoritative transaction or operation id. A term stored only in the scheduler's memory doesn't protect the receiver.

Kill a leader after a committed write, then watch a new term commit with one node down.