LESSON 19.1

Queues, pub/sub and logs

Three shapes of messaging, how a partitioned log like Kafka works, and why replay changes what you can build.

6 min read

"Put it on a queue" covers several different tools. They share the idea of a producer handing work to something else, but they differ in who gets each message, how long messages live and whether you can read them again.

Three shapes

Work queue

Each message to one worker
Per-message acks and retries
Easy to scale workers
Messages vanish once processed
No replay

Pick it when: Background jobs: send this email, resize this image.

Pub/sub

Every subscriber gets a copy
Producers don't know consumers
Classic pub/sub often drops messages for offline subscribers

Pick it when: Broadcasting events to several independent consumers.

Partitioned log

Ordered within a partition
Retained for days, so you can replay
Many independent readers
Parallelism capped by partition count
Offsets, not per-message acks

Pick it when: Event streams that several systems consume, now and later.

Managed services blur the lines (SNS fanning out to SQS queues gives pub/sub with per-subscriber queues), but the three shapes are a useful way to ask what you actually need.

How a log works

Kafka, Kinesis, Pulsar and Redpanda share one model:

A topic is split into partitions. Each partition is an append-only file of messages, each with a sequential offset.
Producers choose a partition, usually by hashing a key such as the order id. Same key, same partition, so order is kept per key.
Partitions are replicated across brokers. A write is acknowledged once enough replicas have it (acks=all with a minimum in-sync count).
Consumers read sequentially and periodically commit the offset they've reached. Messages aren't deleted when read; they expire by retention time or size.

Laying out the diagram…

One log, three consumer groups. Each group reads every event at its own pace.

Consumer groups

A consumer group is a set of instances that share the work of reading a topic. Partitions are divided among them, and each partition goes to one member at a time. If a member dies, its partitions are rebalanced to the others, which resume from the last committed offset.

Different groups are independent: billing, search and analytics each have their own offsets, so a slow analytics loader never holds up billing.

Why replay matters

Because a log keeps messages, new uses don't need new plumbing:

A new service can start from the beginning of retention and build its own view.
A buggy consumer can be fixed and rewound to reprocess the last day.
A search index or cache can be rebuilt from the stream.

That's the core of event-driven architecture, covered further in the events and streams module.

Typical log numbers

Throughput per partitiondepends on message size and disks: ~10 MB/s writes
End-to-end latency: a few ms to tens of ms
Default retentioncan be longer, or forever with tiered storage: 7 days
Partitions per cluster: thousands to tens of thousands

Quick check

You need to send each uploaded image to exactly one of 50 resize workers, retry failures individually, and never need the messages again. Which fits most simply?

Try it

Consumer Group Event Log HLD

Design a partitioned event log with consumer groups and replay.

Check your understanding

1.A message in a classic work queue (SQS, RabbitMQ) is processed successfully. What normally happens to it?
2.Three services each need every order event. In Kafka, how do you give each one the full stream?
3.A topic has 6 partitions. You start 10 consumers in one group. How many actively consume?

Practice it

Consumer Group Event Log HLDDesign retained partition logs with fenced consumer assignments and committed offsets.Distributed Work Queue HLDDesign durable enqueue, visibility leases and stale-ack protection.

In the handbook

KafkaA distributed, partitioned, replicated log. How topics, partitions, replication and consumer groups work, the numbers to quote, and the design decisions that matter.Apache FlinkA stateful stream-processing engine for event-time windows, keyed aggregations and exactly-once guarantees. How watermarks, checkpoints and backpressure work, and when to choose Flink over Kafka Streams or Spark Streaming.SQS and RabbitMQMessage queues for point-to-point task distribution. Visibility timeouts, consumer acknowledgments, dead-letter queues, RabbitMQ exchange routing, and choosing between queues and logs.

Your notes

19. Messaging and logs

LESSON 19.1

Queues, pub/sub and logs

Three shapes of messaging, how a partitioned log like Kafka works, and why replay changes what you can build.

6 min read

Three shapes

Work queue

Each message to one worker
Per-message acks and retries
Easy to scale workers
Messages vanish once processed
No replay

Pick it when: Background jobs: send this email, resize this image.

Pub/sub

Every subscriber gets a copy
Producers don't know consumers
Classic pub/sub often drops messages for offline subscribers

Pick it when: Broadcasting events to several independent consumers.

Partitioned log

Ordered within a partition
Retained for days, so you can replay
Many independent readers
Parallelism capped by partition count
Offsets, not per-message acks

Pick it when: Event streams that several systems consume, now and later.

Managed services blur the lines (SNS fanning out to SQS queues gives pub/sub with per-subscriber queues), but the three shapes are a useful way to ask what you actually need.

How a log works

Kafka, Kinesis, Pulsar and Redpanda share one model:

A topic is split into partitions. Each partition is an append-only file of messages, each with a sequential offset.
Producers choose a partition, usually by hashing a key such as the order id. Same key, same partition, so order is kept per key.
Partitions are replicated across brokers. A write is acknowledged once enough replicas have it (acks=all with a minimum in-sync count).
Consumers read sequentially and periodically commit the offset they've reached. Messages aren't deleted when read; they expire by retention time or size.

Laying out the diagram…

One log, three consumer groups. Each group reads every event at its own pace.

Consumer groups

Different groups are independent: billing, search and analytics each have their own offsets, so a slow analytics loader never holds up billing.

Why replay matters

Because a log keeps messages, new uses don't need new plumbing:

A new service can start from the beginning of retention and build its own view.
A buggy consumer can be fixed and rewound to reprocess the last day.
A search index or cache can be rebuilt from the stream.

That's the core of event-driven architecture, covered further in the events and streams module.

Typical log numbers

Throughput per partitiondepends on message size and disks: ~10 MB/s writes
End-to-end latency: a few ms to tens of ms
Default retentioncan be longer, or forever with tiered storage: 7 days
Partitions per cluster: thousands to tens of thousands

Quick check

You need to send each uploaded image to exactly one of 50 resize workers, retry failures individually, and never need the messages again. Which fits most simply?

Try it

Consumer Group Event Log HLD

Design a partitioned event log with consumer groups and replay.

Check your understanding

1.A message in a classic work queue (SQS, RabbitMQ) is processed successfully. What normally happens to it?
2.Three services each need every order event. In Kafka, how do you give each one the full stream?
3.A topic has 6 partitions. You start 10 consumers in one group. How many actively consume?

Practice it

In the handbook

Your notes