LESSON 10.3
Ordering, priorities and backpressure
When message order matters and how to keep it, separating urgent from bulk work, and slowing producers when consumers can't keep up.
6 min read
Once work flows through queues, three practical questions come up in nearly every design: does order matter, does some work matter more than other work, and what happens when consumers can't keep up?
Ordering
A single queue with many workers processes messages roughly in order, but not exactly: two workers can finish messages in either order, and retries move messages back. Often that's fine. When it isn't (account balance updates, edits to a document, a chat conversation), there are two tools:
- Partition by key. Logs like Kafka keep strict order within a partition. Send all messages for one account, document or conversation to the same partition with a key, and give each partition one consumer. Different keys still run in parallel.
- Make handlers order-tolerant. Include a version or timestamp in each message and ignore anything older than what's already applied. Then order stops mattering.
Priorities and isolation
A password reset email shouldn't wait behind a million newsletter emails. Separate work by urgency:
- Separate queues for urgent and bulk work, each with its own workers, is the simplest and most predictable approach.
- Per-tenant fairness: one big customer's batch shouldn't starve everyone else. Use per-tenant queues or round-robin across tenants.
- Delayed and scheduled jobs ("send in 1 hour", "retry at 10:05") need a queue or scheduler that supports a visible-after time.
Backpressure and load shedding
If producers keep adding work faster than consumers finish it, something has to give. Better to decide what gives than to find out at 3 a.m.
Scale consumers
- Keeps all the work
- Takes time; downstream systems may not cope
Pick it when: The bottleneck is worker capacity, and downstream has room.
Backpressure
- Protects the system
- Producers know to slow down
- Callers must handle rejections (429, blocking)
Pick it when: Sustained overload from controllable producers.
Shed low-priority work
- Keeps important work fast
- Some work is dropped or delayed a lot
Pick it when: Overload where some work genuinely matters less.
Also watch what your workers call. Adding 50 workers to drain a backlog faster can overload the database or the third-party API they talk to. Cap worker concurrency to what downstream can take, or rate-limit their calls.
Workers in practice
- Graceful shutdown: on deploy, stop taking new messages and finish (or release) the current ones.
- Heartbeats for long jobs, extending the visibility timeout so other workers don't grab the job.
- Small, resumable steps for very long jobs (a video transcode, a data export), saving progress so a crash doesn't restart from zero.
Try it
Distributed Job Scheduler HLD
Design a scheduler with delayed jobs, retries, priorities and fairness.