LESSON 18.1
Locks, leases and fencing
Ownership with expiry, the paused-client failure story, receiver-enforced fencing, and the assumptions behind Redis locks and Redlock.
7 min read
A lease makes ownership expire if a worker crashes, loses contact or pauses. Expiry restores progress without waiting for an unlock, but creates a stale-owner problem when the original worker resumes.
Acquire, renew, release
A lease service grants ownership only if the resource is free, returns an owner identity and expiry, and allows renewal while that identity still owns the lease. Release must compare the owner identity atomically; an unconditional delete can release a replacement owner's lease.
For an efficiency-only Redis lease, the basic acquire operation is SET key random-owner NX PX 30000, with an atomic compare-and-delete on release. Renewal must also check ownership. This can reduce duplicate cache rebuilds, but asynchronous failover can lose the lease record and allow another owner.
A random owner identity isn't an increasing fencing token. A client-side deadline says when to stop working, but can't prevent delayed requests reaching the resource.
The GC-pause story
- Worker A acquires a 30-second lease and reads a shared file.
- A pauses for 60 seconds during garbage collection or process suspension.
- The lease expires. B acquires it and updates the file.
- A resumes and writes the old contents, overwriting B's update.
The lease service may have behaved correctly throughout: it granted non-overlapping leases according to its own rules. A simply kept running after losing ownership. Increasing the lease duration reduces the frequency of this story but doesn't make arbitrary pauses or network delays impossible.
Fence at the resource
Each ownership grant includes a strictly increasing fencing generation, say 81 for A and 82 for B. Every protected mutation carries it. The resource atomically tracks the highest accepted generation and rejects older ones, so after accepting B's 82 it rejects A's delayed 81.
Issuance must remain ordered across failures. Enforcement must be atomic with the effect, not a separate "check token" request followed by an unguarded write. For workflows requiring immediate revocation, make advancing the resource's generation part of ownership activation before the replacement starts work.
A downstream API that ignores generations can't be protected merely by attaching a token. Use provider idempotency, a transactional authoritative service, or a workflow designed to tolerate repeated effects. Even a current generation doesn't deduplicate two requests from the same owner.
The Redlock debate
Redlock tries to obtain short-lived locks on a majority of independent Redis instances within a bounded acquisition time. Its reasoning depends on timing and clock-drift assumptions. It isn't the same thing as a consensus-replicated lock service.
Redis documents the algorithm and competing analyses. Kleppmann's critique emphasizes pauses, delayed requests and missing fencing; its defenders argue about the intended assumptions and use cases. In an interview, identify the failure model and the cost of duplicate work rather than treating the name as a safety guarantee.
Efficiency lock
- Reduces redundant work
- Can stay simple
- Occasional duplicate execution must be harmless
Pick it when: Coalescing cache refreshes or repeated calculations.
Correctness-critical ownership
- Can protect an invariant with an authoritative protocol
- Requires fenced effects or transactional checks
- May stop during loss of quorum
Pick it when: Only one owner may mutate a shared resource safely.
Try it
Distributed Lock Service HLD
Design lease acquisition and release, then show how the protected resource rejects stale owners.