Practical coordination

A coordination service records small shared decisions: endpoint registrations, active configuration and partition ownership. Keep those explicit, then handle ordinary requests using local state wherever requirements allow.

Step through gossip rounds, then introduce packet loss and inspect failure suspicions.

Service discovery

An instance registers its address, service name and metadata, tied to a lease or session. It renews while healthy; clean shutdown deregisters it, while expiry handles crashes. Clients watch registrations or resolve them through a discovery layer, then cache the endpoint set locally.

Registration doesn't prove health: an instance may be unable to reach its database. A failed renewal may mean the registry is unreachable. Combine registrations with health checks, connection failures and bounded retries; avoid a registry call on every request.

Configuration as a versioned decision

Store immutable configuration versions and a small active-version pointer. Writers validate the proposed configuration, then conditionally update the pointer against the expected revision so two operators can't silently overwrite each other's changes. Clients validate again before activating locally.

An atomic pointer change doesn't update every process simultaneously. Use staged rollouts, compatibility windows and acknowledgement reporting. Rollback is another versioned change; software must tolerate the versions it may encounter.

Watches need recovery

A watch is a stream, not a promise that a client never disconnects. Join a snapshot to changes at its revision, keep the last applied revision, and resume after reconnect. If retained history no longer covers it, take a new snapshot and restart the watch without a gap.

Apply updates in order, ignore applied revisions, and track lag. Watch events update local state; they aren't perpetual permission to mutate resources. Enforce ownership generations at the receiver.

Membership isn't exclusive authority

Gossip can spread membership and failure suspicions cheaply across many nodes. Different nodes may temporarily disagree, which is acceptable for routing hints and load estimates. It's not enough by itself to assign one correctness-critical owner: both sides of a partition can suspect the other and promote themselves.

Separate failure detection from authority. A timeout can trigger an election attempt, but only the agreed protocol grants ownership. Delayed work from a removed member remains possible, so apply the fencing rules from the previous lessons.

Avoid coordination where possible

Adding 3 and adding 5 commute, but receiving "add 3" twice still adds 6. Setting a flag to true is idempotent, but setting true and setting false don't commute. Pick the property actually needed; don't treat the terms as synonyms.

A practical scheduler coordinates partition ownership, then handles many jobs locally under that generation. Moving ownership is infrequent; every job doesn't need a global election. Keep cross-owner effects idempotent and identify the few invariants that still require shared authority.