Technology · Event log
Kafka
A distributed, partitioned, replicated log. How topics, partitions, replication and consumer groups work, the numbers to quote, and the design decisions that matter.
11 min read
Kafka stores streams of events as ordered, replicated logs and lets many consumers read them independently, at their own pace, and again later. It's the default backbone for event-driven systems, change data capture, metrics and log pipelines.
When to reach for it
- Several services need the same stream of events.
- You want to replay: rebuild a search index, backfill a new service, reprocess after a bug.
- Throughput is high: hundreds of thousands to millions of events per second.
- Order matters per key, such as all events for one order in sequence.
For a simple job queue with per-message retries and a handful of workers, SQS or RabbitMQ is usually simpler.
The model
Laying out the diagram…
- A topic is split into partitions, each an append-only log with sequential offsets.
- Producers pick a partition by hashing the message key. Same key, same partition, so per-key order holds.
- Each partition has a leader broker and follower replicas. With
acks=allandmin.insync.replicas=2, a write is acknowledged once two replicas have it. - Consumers in a group split the partitions between them and commit offsets. Different groups are independent.
- Messages are deleted by retention (time or size), not by being read. Compacted topics keep the latest value per key.
Modern Kafka manages cluster metadata with its own Raft quorum (KRaft) instead of ZooKeeper.
Numbers to quote
- Throughput per partitionset partitions from target throughput
- ~5–10 MB/s
- Cluster throughput
- GB/s with a few dozen brokers
- End-to-end latencybatching trades latency for throughput
- ~5–50 ms
- Retentiontiered storage makes months cheap
- 7 days default
- Max message sizestore big payloads in S3, send a pointer
- 1 MB default
Key design decisions
Partition key
- Defines ordering and load spread
- Skewed keys create hot partitions
Pick it when: Key by the entity that needs ordering: order id, user id, device id.
Partition count
- More partitions, more parallel consumers
- Adding later remaps keys
- Very high counts add overhead
Pick it when: Enough for peak consumer parallelism, with 2–3× headroom.
Durability settings
- acks=all + min ISR 2 + replication 3 survives a broker loss
- Slightly higher write latency
Pick it when: Anything you can't lose.
Delivery semantics
- Commit offsets after processing for at-least-once, and make consumers idempotent.
- Idempotent producers remove duplicates from producer retries.
- Transactions atomically write output topics and commit input offsets, giving exactly-once processing for Kafka-to-Kafka pipelines. Side effects outside Kafka still need idempotency keys.
Where it bites
- Rebalances pause consumption; use cooperative rebalancing and static membership.
- It's operationally heavy to run yourself. Managed offerings (MSK, Confluent Cloud) are common answers.
- It's not a database: no queries by field, no updates in place.
Try it
Consumer Group Event Log HLD
Design a partitioned log with consumer groups and replay.
Learn the ideas
Practice it
Consumer Group Event Log HLDDesign retained partition logs with fenced consumer assignments and committed offsets.Log Aggregation Platform HLDCollect tenant-scoped logs with durable ingestion, searchable indexes and retention.Change Data Capture Pipeline HLDBridge a consistent snapshot to a retained change log and replay downstream safely.Metrics Ingestion HLDIngest time-series data with bounded cardinality, replay and predictable query costs.
Related
In-memory storeRedisAn in-memory data structure server used as a cache, rate limiter, leaderboard, lock service, queue and pub/sub hub. What it's good at, how it scales, and where it bites.Search engineElasticsearchA distributed search and analytics engine built on inverted indexes. Indexing and analysis, relevance, shards and replicas, keeping it in sync, and running it well.Stream processorApache FlinkA stateful stream-processing engine for event-time windows, keyed aggregations and exactly-once guarantees. How watermarks, checkpoints and backpressure work, and when to choose Flink over Kafka Streams or Spark Streaming.Message queueSQS and RabbitMQMessage queues for point-to-point task distribution. Visibility timeouts, consumer acknowledgments, dead-letter queues, RabbitMQ exchange routing, and choosing between queues and logs.Columnar storeClickHouseA fast columnar analytical database for real-time aggregations over billions of events. The MergeTree engine family, columnar compression, vectorized execution, and batch ingestion.ScalingScaling writesWhen writes outgrow one machine. Make writes cheaper, batch and buffer them, partition them, and pick a storage engine built for them.WorkflowsMulti-step processesBusiness processes that span services, minutes or days, such as orders, onboarding and payouts. State machines, sagas, durable workflow engines, timeouts and human steps.ConsistencyEvent Sourcing and CQRSTreating an append-only sequence of domain events as the authoritative system of record. Read and write decoupling with Command Query Responsibility Segregation (CQRS), asynchronous projections, aggregate snapshots, schema evolution upcasters, and operational antipatterns.