Technology · Message queue
SQS and RabbitMQ
Message queues for point-to-point task distribution. Visibility timeouts, consumer acknowledgments, dead-letter queues, RabbitMQ exchange routing, and choosing between queues and logs.
11 min read
Message queues decouple producers from consumers by holding discrete tasks until a worker is ready to execute them. Unlike an event log like Kafka, where messages are immutable sequential log records read by consumer groups, message queues are designed for destructive consumption: each message is pulled, processed, acknowledged, and promptly removed from the queue.
When to reach for a queue
- Work distribution (Competing consumers): Distributing CPU-intensive jobs (video encoding, PDF generation, image resizing) across a fleet of stateless workers.
- Buffer asynchronous RPCs: Throttling requests to slow third-party APIs (payment gateways, SMS providers) so spikes do not overwhelm rate limits.
- Individual message retries: Retrying a specific failed job with exponential backoff without stalling all subsequent messages.
- Complex routing: Routing messages by topic patterns or fanout bindings across multiple independent worker pools.
If you need event replayability, streaming aggregations, or millions of events consumed by multiple independent teams at their own pace, reach for Kafka instead.
Queues vs logs
Message Queue (SQS / RabbitMQ)
- Per-message acknowledgment and redelivery
- Worker pools scale independently of partition counts
- Poison messages move to dead-letter queue without blocking others
- Messages destroyed after consumption
- No replay capability
- Strict total ordering requires single-consumer bottlenecks
Pick it when: Asynchronous task execution, webhooks, background jobs, worker pools.
Event Log (Kafka)
- Messages retained for days or weeks
- Replay events to backfill or rebuild state
- Millions of events/s per cluster
- One slow message blocks its entire partition
- Consumer concurrency strictly limited by partition count
Pick it when: Event sourcing, metrics pipelines, change data capture, real-time analytics.
Amazon SQS: architecture and semantics
Amazon Simple Queue Service (SQS) is a fully managed, serverless distributed queue. You don't manage brokers, clusters, or disk space.
- A producer calls
SendMessage(orSendMessageBatchup to 10 messages). - The message is replicated across multiple availability zones and enters the queue.
- A consumer pulls messages using
ReceiveMessage. SQS does not push; consumers poll, preferably using long polling (WaitTimeSeconds up to 20s) to reduce empty receives and cost. - When pulled, the message enters a visibility timeout (default 30 seconds): it remains in the queue but is invisible to other workers while being processed.
- If the worker succeeds, it calls
DeleteMessage. If the worker crashes or times out before deleting, the visibility timeout expires and the message becomes visible to another worker for retry. - If a message fails repeatedly (exceeding
maxReceiveCount), SQS shifts it to a Dead-Letter Queue (DLQ).
Standard vs FIFO queues
- Standard SQS: Nearly unlimited throughput, at-least-once delivery, and best-effort ordering. Two copies of a message can occasionally be delivered, so consumers must be idempotent.
- FIFO SQS: Guarantees strictly once processing and first-in-first-out ordering within a MessageGroupId. Supports up to 300 transactions/second (or 3,000/s with batching). Deduplication is automatic using a 5-minute hash window on
MessageDeduplicationId.
RabbitMQ: exchanges and flexible routing
RabbitMQ is an open-source message broker implementing the AMQP protocol. Unlike SQS's direct queue model, RabbitMQ separates the message destination into Exchanges and Queues, joined by Bindings.
RabbitMQ provides four exchange types:
Direct Exchange
- Exact routing key match
- No pattern matching
Pick it when: Routing task types (send.email, send.sms) directly to dedicated queues.
Fanout Exchange
- Broadcasts message to every bound queue
- Ignores routing key
- High duplicate amplification
Pick it when: Pub/Sub broadcasts (invalidating local caches on all servers).
Topic Exchange
- Wildcard routing with * (one word) and # (zero or more words)
- Slightly higher routing overhead
Pick it when: Hierarchical event routing (eu.orders.created, us.orders.cancelled).
Headers Exchange
- Routes based on message header attributes rather than routing key string
- Less common, slightly slower
Pick it when: Routing where matching requires multiple metadata attributes.
Consumer flow control: prefetch count
In RabbitMQ, the broker pushes messages over a stateful TCP connection. Without protection, a fast broker will dump thousands of messages into a single worker's memory buffer, causing out-of-memory crashes while peer workers sit idle.
Workers configure basic.qos(prefetch_count = 50): the broker sends at most 50 unacknowledged messages to a worker at any time. When a worker completes a task and calls basic.ack, the broker releases the next message.
Numbers to quote
- SQS Standard throughputscales automatically across AWS partitions
- Virtually unlimited
- SQS FIFO throughputcan scale with unique MessageGroupIds
- 300 req/s unbatched, 3,000 req/s batched
- SQS Latencydue to distributed HTTP polling across zones
- 10–30 ms
- RabbitMQ throughputdepends on disk persistence and message size
- 20,000–100,000 msgs/s per node
- RabbitMQ Latencypersistent TCP connection, push-based delivery
- < 5 ms
- Max message sizestore large payloads in S3 and pass a reference
- 256 KB (SQS)
Choosing the right messaging tool
Amazon SQS
- Zero operations, fully serverless
- Virtually infinite scale on standard queues
- Built-in DLQs and retry counters
- Higher latency (HTTP polling)
- Simple FIFO throughput limits
- AWS lock-in
Pick it when: Cloud-native microservices on AWS, background tasks, webhooks.
RabbitMQ
- Sub-5ms push latency
- Sophisticated AMQP routing and topic bindings
- Runs anywhere (on-prem, k8s, cloud)
- Brokers must be clustered and managed
- RAM and disk pressure require tuning
Pick it when: Complex routing topologies, on-prem deployments, low-latency RPC.
Apache Kafka
- High-throughput replayable log
- Consumer groups read independently
- Built-in partitioning
- Head-of-line blocking per partition
- Operationally heavyweight
Pick it when: Event pipelines, streaming analytics, system-wide event logs.
Where it bites
- Poison pills: A corrupted message triggers an unhandled crash in your worker code. When the visibility timeout expires, another worker picks it up and crashes. Without a Dead-Letter Queue with a tight
maxReceiveCount(e.g. 3 retries), a single bad message can repeatedly crash your entire worker fleet. - Duplicate execution: Networks fail after a worker completes a task but before its acknowledgment reaches the broker. Queues will redeliver the message. Processing must be idempotent using a unique idempotency key or database unique constraint.
- Queue backlog amnesia: Alert on queue age (the oldest in-flight message timestamp), not just queue length. Ten thousand bulk newsletters in a queue is fine; ten password reset emails that are ten minutes old is an outage.
Try it
Notification Service HLD
Build an asynchronous notification pipeline with queues, workers and provider retries.