Stream joins and enrichment

Raw events rarely carry everything a consumer needs. A click has a user id but not a country; a payment has a merchant id but not the merchant's risk tier. Stream processing adds that context with joins, and does so without calling a database for every event.

Enrichment with a stream-table join

The trick is to treat reference data as a stream too. The users table's changes flow through CDC into a topic, and the processor keeps the latest row per user in a local state store (an embedded key-value store such as RocksDB, backed by a changelog topic).

Each click looks up its user locally, in microseconds, and moves on with the country attached. When a user changes country, the CDC event updates the local table, and later clicks see the new value.

Co-partitioning

A join task only sees the partitions assigned to it. For a click to meet its user record, both topics must be partitioned by the same key, with the same number of partitions, using the same hash. If they aren't, the processor first repartitions one stream (writes it back out keyed correctly), which costs an extra hop.

Small reference tables (a few thousand countries or products) skip this: every task gets the whole table, a broadcast or "global" table.

Stream-stream joins

Sometimes both sides are events: match each ad impression with the click that followed it, or each order with its payment. Neither side is a table, so the join needs a window: "join a click to an impression with the same id within 30 minutes". The processor buffers both sides for the window's length, and late or unmatched events need a policy (emit unmatched after the window, or drop them).

Evolving schemas

Events outlive the code that wrote them, and many consumers read each topic. Changing a field is a contract change: