Pattern · Workflows
Multi-step processes
Business processes that span services, minutes or days, such as orders, onboarding and payouts. State machines, sagas, durable workflow engines, timeouts and human steps.
11 min read
An order is placed, paid, packed, shipped and delivered. A driver is matched, arrives, completes a trip and gets paid. A new customer signs up, verifies identity, is reviewed by a person and is activated. Each of these is a multi-step process: several services, external calls that fail, waits of minutes or days, and the need to always know where each instance stands.
The problem
Writing it as a chain of calls inside one request fails as soon as any step is slow or a process restarts. Writing it as services reacting to each other's events works for two steps, but with ten the flow is invisible, retries are inconsistent, and nobody can answer "why is order 812 stuck?".
Make the state explicit
Model each process as a state machine stored durably:
- Define states (CREATED, PAID, PACKED, SHIPPED, DELIVERED, CANCELLED) and the allowed transitions.
- Store the current state, a version and timestamps on the process's record.
- Move between states with conditional updates (
WHERE state = 'PAID'), so duplicate events and races can't skip or repeat a transition. - Emit an event on each transition (through an outbox) for other services and for audit.
Ways to drive it
Choreography
- No central component
- Services stay decoupled
- Flow is spread across services
- Hard to change and debug
Pick it when: Two or three steps that rarely change.
Orchestrator service + state table
- Flow in one place
- Easy to see each instance's state
- You build retries, timers and recovery yourself
Pick it when: Moderate complexity, team prefers plain services.
Durable workflow engine
- Workflow code survives crashes
- Built-in retries, timers, history
- New infrastructure and programming model
Pick it when: Many steps, long waits, human tasks: Temporal, Step Functions, Cadence.
Durable workflow engines let you write the process as ordinary code (charge(), then await sleep(3 days), then remind()), while the engine records every step's result. If a worker dies, another replays the history and continues where it stopped.
Failure, timeouts and compensation
- Every step is idempotent, called with a stable key derived from the process id and step, because every step will sometimes run twice.
- Retry transient failures with backoff; give each step a deadline.
- Compensate when a later step fails permanently: refund the payment, release the stock. Order steps so irreversible ones come last.
- Timeouts are business events: "payment not confirmed in 15 minutes" moves the order to CANCELLED and releases its hold.
- Unknown outcomes (a timed-out call that may have succeeded) are resolved by querying the other service with the idempotency key before deciding.
Human steps and long waits
Approvals, manual reviews and "wait for the customer to upload a document" can take days. Model them as states with deadlines and reminders, not as threads that block. A workflow engine's durable timers, or a scheduler that scans for overdue states, handles this cleanly.
Try it
Durable Workflow Orchestrator HLD
Design a workflow engine that runs processes to completion through crashes.