Design a distributed job scheduler

Advanced60 minBuilds Distributed Job SchedulerRead first: 10.1 Why queues, 10.2 Delivery guarantees and idempotent consumers, 17.1 Timeouts and retries

STEP 1 OF 13

The brief

You are designing a distributed background job scheduler. The system accepts millions of delayed, recurring, and asynchronous tasks from upstream microservices, persists them durably, and ensures they execute at their scheduled timestamps.

Building a dependable distributed scheduler introduces critical concurrency and reliability challenges:

At-least-once execution: Workers crash mid-job, networks partition, and nodes reboot. Jobs must never be lost, even if an execution attempt fails midway.
Idempotency: Because failures cause retries, jobs may run more than once. Side effects must be safeguarded with deduplication and idempotency keys.
Leases and fencing: When a worker stalls on a long GC pause, a supervisor may reassign its job to another worker. Fencing tokens prevent the zombie worker from corrupting state upon waking up.
Scheduler coordination: Multiple scheduler instances must scan due jobs without creating duplicate queue dispatches or lock contention.

All changes saved

0 components · 0 connections

A blank canvas. A better system.

Add components from the palette, then drag between them to connect the request and event flow.

React Flow

Design a distributed job scheduler

Advanced60 minBuilds Distributed Job SchedulerRead first: 10.1 Why queues, 10.2 Delivery guarantees and idempotent consumers, 17.1 Timeouts and retries

STEP 1 OF 13

The brief

Building a dependable distributed scheduler introduces critical concurrency and reliability challenges:

At-least-once execution: Workers crash mid-job, networks partition, and nodes reboot. Jobs must never be lost, even if an execution attempt fails midway.
Idempotency: Because failures cause retries, jobs may run more than once. Side effects must be safeguarded with deduplication and idempotency keys.
Leases and fencing: When a worker stalls on a long GC pause, a supervisor may reassign its job to another worker. Fencing tokens prevent the zombie worker from corrupting state upon waking up.
Scheduler coordination: Multiple scheduler instances must scan due jobs without creating duplicate queue dispatches or lock contention.

All changes saved

0 components · 0 connections

A blank canvas. A better system.

Add components from the palette, then drag between them to connect the request and event flow.

React Flow