Pattern · Async work
Long-running tasks
Work that takes seconds to hours, such as video transcoding, exports and report generation. Accept fast, process in workers, report progress, and survive crashes and retries.
10 min read
Some requests can't finish within an HTTP timeout: transcoding a video, generating a 2 GB export, importing a million rows, training a model, sending a campaign to every user. Holding the request open wastes a server, breaks on any network blip and gives the user nothing to look at. The pattern is to split accepting the work from doing it.
The shape
- The API validates the request, creates a job record (id, type, input, status PENDING, owner) and enqueues the job id. It returns 202 Accepted with the job id.
- A pool of workers takes jobs from the queue, marks them RUNNING, and does the work.
- Workers write results somewhere durable, usually object storage for big outputs, then mark the job SUCCEEDED or FAILED.
- The client learns the outcome by polling
GET /jobs/:id, by a push (WebSocket, SSE, email), or by a webhook for API customers.
Write the job record and the queue message reliably together, for example with an outbox, so a job can't be recorded but never queued.
Surviving failure
Visibility timeout + heartbeats
- A crashed worker's job reappears for another worker
- A slow but alive worker must extend its lease
Pick it when: Every queue-based worker.
Idempotent steps
- Retries and duplicates are harmless
- Needs stable ids for outputs
Pick it when: Always: jobs will run twice sometimes.
Checkpointing
- Resume from the last saved point, not from zero
- State to store and design
Pick it when: Jobs longer than a few minutes.
Retries with backoff, then dead letter
- Transient errors heal; bad jobs stop looping
- Needs a place and process for dead jobs
Pick it when: Every job type.
Splitting big jobs
A 3-hour job that fails at 2h59m is painful. Break it up:
- Chunk the input (rows 1–10,000, 10,001–20,000…) into many small jobs that run in parallel, then a final job combines results.
- Track chunk completion in the job record so progress is real: "38 of 120 parts done".
- For video, transcode segments in parallel and stitch the manifest at the end.
Capacity and fairness
- Autoscale workers on queue depth or age, not CPU.
- Separate queues for interactive jobs (a user is waiting) and bulk jobs, so a nightly batch never delays a user's export.
- Cap per-tenant concurrency so one customer's 10,000 jobs don't starve everyone.
- Protect downstreams: 500 workers can overwhelm a database that 50 would not.
- Exports requested
- 2,000 per hour at peak
- Average duration
- 90 s
- Concurrent jobs needed
- 2,000 × 90 ÷ 3,600 = 50
- Workers with headroom
- ~75
Try it
Large Data Export Platform HLD
Design large data exports that users can request and download.