LESSON 20.4
Durable workflows
Workflow engines like Temporal and Step Functions, how they persist progress and replay code deterministically, timers that last days, waiting for people, and versioning running workflows.
6 min read
Sagas describe what a multi-step business process does. Running one reliably for hours or days, through crashes, deploys and retries, is a lot of plumbing: state tables, queues, timers, cron jobs to find stuck instances. Durable workflow engines package that plumbing.
The idea
You write the process as ordinary-looking code:
- Reserve inventory.
- Charge the card.
- Wait up to 3 days for the warehouse to confirm.
- If confirmed, ship; otherwise refund and release inventory.
The engine (Temporal, AWS Step Functions, Azure Durable Functions, Restate and others) records every step's result in a durable history. If the process running the workflow dies, another worker picks it up and continues from the last recorded step. A "wait 3 days" is a durable timer, not a sleeping thread.
Workflows and activities
Engines separate two kinds of code:
Workflow code
- Durable, resumable
- Can wait for days
- Must be deterministic: no direct I/O, clock or randomness
Pick it when: The orchestration logic: order of steps, branching, waiting, compensating.
Activities
- Any code: API calls, database writes
- At-least-once; may run more than once
Pick it when: Every side effect.
In Temporal-style engines, recovery works by replay: the workflow code runs again from the top, but each activity call returns its recorded result instead of executing. That's why workflow code must make the same decisions every time; it gets time, random values and side effects only through the engine.
Timers and signals
Two features make workflows different from a job queue:
- Durable timers. "Remind the user in 24 hours, cancel the reservation after 15 minutes, escalate after 3 days." Thousands of timers per second, surviving restarts.
- Signals. The workflow waits for an external event: a warehouse confirmation, a human approval, a webhook. Combined with a timer, this gives "wait for approval, but escalate if nobody approves within two days" in a few lines.
What it doesn't solve
Two more practical issues:
- Versioning. A workflow started last week is still running old code's history when you deploy new code. Changing the order of steps can break replay. Engines provide versioning markers so old executions follow old logic.
- History size. Every step adds to the history. Workflows that loop forever (a subscription that bills monthly for years) periodically restart themselves with fresh history ("continue as new").
When to use one
Use a workflow engine when a process has several steps across services, waits for time or people, and must never get lost. For a single background job, a queue is simpler. For a two-step process inside one database, a transaction is simpler still.
Try it
Durable Workflow Orchestrator HLD
Design an engine that runs long-lived workflows reliably.