LESSON 21.1
Change data capture and derived data
Stream every committed change out of a database, and use it to keep caches, search indexes and warehouses in sync without dual writes.
5 min read
Most large systems keep the same data in several shapes: rows in Postgres, documents in a search index, entries in a cache, tables in a warehouse. The question is how to keep them in sync. Change data capture (CDC) answers it by turning the database's own commit log into a stream.
Systems of record and derived data
- The system of record holds the authoritative copy. Writes go here first.
- Derived data is computed from it: caches, search indexes, materialized views, analytics tables, recommendations.
Derived data can always be thrown away and rebuilt. Keeping that property is what makes a design easy to evolve.
How CDC works
- A connector (Debezium, or a managed equivalent) reads the database's replication log: the WAL in Postgres, the binlog in MySQL.
- Each committed insert, update or delete becomes an event with the key, the before and after values, and the log position.
- Events go to a log, one topic per table, keyed by primary key so each row's changes stay in order.
- Consumers apply them to their own stores, idempotently, tracking the position they've reached.
Why it beats dual writes
App writes to every store
- Simple to start
- Crash between writes leaves stores inconsistent
- Concurrent writes can apply in different orders
Pick it when: Prototypes only.
CDC from the database
- Only committed changes, in commit order
- No app changes
- New consumers can be added later
- Another moving part
- Events mirror tables, not business intent
Pick it when: Keeping derived stores in sync.
Outbox
- Events express business intent
- Schema you control
- App must write outbox rows
Pick it when: Events other teams consume as a contract.
Operational notes
- Derived stores lag the source, typically by milliseconds to seconds. Read-your-writes flows may need to read from the source.
- A replication slot that isn't consumed holds WAL on the primary, which can fill its disk. Monitor slot lag.
- Schema changes on the source flow downstream; handle them like any message schema change.
Try it
Change Data Capture Pipeline HLD
Design a CDC pipeline from an OLTP database to several consumers.