LESSON 19.4
Choosing a broker
Kafka, RabbitMQ, SQS, cloud pub/sub and Redis Streams compared by model, ordering, retention and operations, plus sizing partitions and retention for a real workload.
5 min read
"Use a message queue" is rarely the whole answer. The brokers on the market make different promises, and picking one is a design decision you should be able to justify.
The main options
Kafka / Kinesis / Redpanda (log)
- Very high throughput
- Retention and replay
- Many consumer groups
- Order per partition
- Partition count limits parallelism
- No per-message retry or delay built in
Pick it when: Event streams, CDC, analytics pipelines, fan-out to many services.
RabbitMQ (broker with routing)
- Flexible routing (topic, fanout)
- Per-message ack, priorities
- Messages gone once consumed
- Throughput lower than logs
Pick it when: Task distribution with routing rules, request/reply patterns.
SQS / Cloud Tasks (managed queue)
- Nothing to run
- Visibility timeouts, delays, dead letters
- Limited ordering (FIFO queues are slower)
- No replay
Pick it when: Background jobs and work queues in the cloud.
SNS / Google Pub/Sub (managed pub/sub)
- Push to many subscribers
- Scales automatically
- Short retention, limited replay
Pick it when: Fan-out notifications between services.
Redis Streams
- Already in many stacks
- Consumer groups, low latency
- Memory-bound retention
- Weaker durability
Pick it when: Moderate volumes where Redis is already trusted.
A common combination: a log (Kafka) as the backbone for domain events, with managed queues for per-task work that needs retries and delays.
Questions that decide
- Does more than one consumer need each message? Then pub/sub or a log, not a single queue.
- Must anyone replay history? Only a log with retention does that well.
- Does order matter? Logs order within a partition; FIFO queues order within a group at lower throughput.
- Do individual messages need retries and delays? Queues have these built in; on a log you build retry topics.
- Who will run it? A managed service often beats a better-fitting system nobody on the team can operate.
Sizing a log
For a log, three numbers matter: throughput, partition count and retention.
- Peak rate
- 20,000 events/s × 1 KB = 20 MB/s
- Consumer speed
- about 2,000 events/s per instance - so 10 consumers now
- Partitions
- about 30 - room to triple consumers without repartitioning
- Retention
- 7 days × 20 MB/s × 86,400 s ≈ 12 TB, × 3 replicas ≈ 36 TB
Partitions are hard to change later: adding them changes which partition a key hashes to, breaking per-key order for a while. So pick a number that leaves room for growth, without going to thousands, since each partition has overhead.
Delivery and operations
Whichever you choose, the earlier lessons still apply: assume at-least-once delivery, make consumers idempotent, watch consumer lag or queue age, and give poison messages somewhere to go. The broker changes how these are implemented, not whether you need them.
Try it
Notification Service HLD
Pick and justify the messaging system for notifications.