LESSON 10.4
Scheduled and long-running jobs
Delayed and recurring jobs without a single cron box, splitting work that takes hours into resumable pieces, heartbeats and progress, and cancelling work safely.
6 min read
Queues are perfect for work that should happen soon. Two other kinds of background work need more: work that should happen at a certain time, and work that takes a long time.
Delayed and recurring jobs
"Send a reminder in 24 hours" and "run billing every night at 02:00" are both timers. Options, roughly in order of scale:
Queue with delay
- No new infrastructure
- Short maximum delays (SQS: 15 minutes)
- Hard to cancel or list
Pick it when: Retries with backoff, short delays.
Jobs table polled by workers
- Durable, queryable, cancellable
- Works with any database
- Polling load
- Needs careful locking
Pick it when: Most apps, up to thousands of due jobs a second.
Dedicated scheduler
- Partitioned by time and id
- Scales to millions of timers
- A system of its own to run
Pick it when: Very large numbers of timers or strict timing.
A jobs table is the common middle ground. Workers claim due rows with SELECT ... WHERE run_at <= now() ... FOR UPDATE SKIP LOCKED, which lets many workers pull different rows without blocking each other. For recurring jobs, the scheduler computes the next run_at when it enqueues the current run, and uses a unique key like (job, scheduled_time) so two schedulers can't create the same run twice.
Long-running work
A job that runs for hours will be interrupted: by a deploy, an autoscaler removing the machine, a spot instance being reclaimed or a crash. Design for it:
- Split. Break the work into pieces that each take seconds to minutes: one per video segment, per thousand rows, per customer. A parent job tracks which pieces are done.
- Checkpoint. If a job can't be split, save progress periodically (a cursor, an offset) so a retry resumes rather than restarts.
- Make pieces idempotent. A piece may run twice when a worker dies after finishing but before recording it. Writing output to a deterministic location makes the second run harmless.
Heartbeats, progress and cancellation
A visibility timeout long enough for a two-hour job also means a crashed worker's job sits stuck for two hours. Instead, use a short lease that the worker extends with heartbeats. When heartbeats stop, the job is reassigned within seconds.
The same heartbeat can carry progress ("segment 31 of 48"), which drives a progress bar and spots jobs that are alive but stuck. And it gives a natural place for cancellation: the reply to a heartbeat can say "stop", and the worker cleans up and exits at the next safe point.
Try it
Distributed Job Scheduler HLD
Design a scheduler that never loses a due job and never runs one twice by accident.