LESSON 21.3
Stream processing
Computing over unbounded streams with windows, the difference between event time and processing time, watermarks, late data, and fault-tolerant state.
6 min read
Batch jobs compute over a finished dataset: yesterday's logs. Stream processing computes continuously over data that never ends: clicks per minute right now, fraud scores on every payment, a live leaderboard. Flink, Kafka Streams, Spark Structured Streaming and cloud equivalents all share the ideas below.
Windows
An unbounded stream needs boundaries to aggregate over:
Tumbling
- Simple, non-overlapping
- Bursts split across boundaries
Pick it when: Counts per minute, per hour.
Sliding (hopping)
- Smooth, overlapping views
- Each event lands in several windows
Pick it when: Last 5 minutes, updated every 30 seconds.
Session
- Matches real user activity
- Variable length, harder to bound state
Pick it when: User sessions, bursts of device activity.
Event time versus processing time
Every event has two times: event time, when it happened, and processing time, when the processor sees it. They differ because of network delay, offline devices, retries and backlogs.
- Mobile events delayed by offline phones
- minutes to days
- Consumer catching up after an outage
- an hour of events processed in minutes
- Effect of using processing time
- a traffic spike that never happened
Windowing by event time gives correct answers even when data arrives out of order, but raises a question: when is a window complete?
Watermarks and late data
A watermark is the processor's running estimate that all events up to time T have arrived. When the watermark passes a window's end, the window fires.
- Watermarks typically trail the highest event time seen by a bound, such as 10 seconds of allowed out-of-orderness.
- Events older than the watermark are late. Policies: drop them, send them to a side output, or allow lateness and emit an updated result.
- Downstream sinks must accept updates (upserts) if windows can re-fire.
A tight watermark gives fast but less complete results; a loose one is more complete but slower. Pick based on what the result is for.
State and fault tolerance
Aggregations, joins and deduplication keep state: counts per window, the last value per key. Processors keep it in local storage (often RocksDB), partitioned by key like the input, and periodically checkpoint it together with input offsets to durable storage.
Restoring state and offsets together gives exactly-once state. Exactly-once output also needs a transactional or idempotent sink.
Try it
Event-time Stream Aggregation HLD
Aggregate events by event time with watermarks and late data.