LESSON 19.3
Running a log in production
Picking keys and partition counts, watching consumer lag, surviving rebalances, poison messages, schema changes and log compaction.
5 min read
The concepts of a log are simple. Most operational pain comes from a handful of decisions: keys, partition counts, slow or failing consumers, and changing message formats.
Keys and hot partitions
The partition key decides ordering and load. Key by the entity whose events must stay in order: order id, account id, device id. Watch for skew: if one customer produces 30% of events, keying by customer puts 30% of load on one partition and one consumer.
When a hot key appears, the options mirror database hot keys: key by a finer entity, add a suffix to spread one key across several partitions (giving up order for it), or give the heavy tenant its own topic.
Consumer lag
- Produce rate
- 20,000 msgs/s
- Consume rate
- 15,000 msgs/s
- Lag growth300,000 per minute
- 5,000 msgs/s
- Time to drain 1M lag at 25,000/s consumeonly the surplus over produce rate drains it
- 1,000,000 / 5,000 = 200 s
Alert on lag that grows steadily, or on lag expressed as time (how old the oldest unread message is). Fix it by adding consumers (up to the partition count), batching, or speeding up the downstream they write to.
Rebalances
When a consumer joins, leaves or misses heartbeats, the group reassigns partitions. During an eager rebalance, consumption pauses. To keep them rare and short:
- Keep processing per poll short, so consumers don't miss their poll deadline and get kicked out.
- Use cooperative (incremental) rebalancing and static membership so a rolling deploy doesn't reshuffle everything.
- Commit offsets before giving up a partition.
Poison messages and dead letters
- Retry a failing message a few times with backoff.
- If it still fails, publish it to a dead-letter topic with the error and move on.
- Alert on the dead-letter topic, fix the bug, and replay its messages.
For transient downstream outages, a retry topic with a delay works better than blocking the partition.
Changing message formats
Producers and consumers deploy at different times, and a log may hold week-old messages. Treat message schemas like an API:
- Use a schema format (Avro, Protobuf, JSON Schema) and a schema registry that rejects incompatible changes.
- Add optional fields; never reuse or change the meaning of existing ones.
- Version breaking changes as a new topic or event type, and run both during migration.
Try it
Log Aggregation Platform HLD
Collect, buffer and index logs from thousands of services.