Technology · Stream processor
Apache Flink
A stateful stream-processing engine for event-time windows, keyed aggregations and exactly-once guarantees. How watermarks, checkpoints and backpressure work, and when to choose Flink over Kafka Streams or Spark Streaming.
11 min read
Flink processes unbounded streams of events and keeps state across them. Unlike a batch job that waits for all the data, Flink answers continuously: the one-minute fraud score for every account updates every second, using the last hour of transactions as context. It's the engine behind real-time dashboards, fraud detection, top-K rankings and metrics pipelines at companies where Kafka Streams hits its limits.
When to reach for it
- You need keyed aggregations over time windows: revenue per merchant in the last five minutes, error rate per service in the last hour.
- You care about event time, not when the data arrived. Mobile events arrive late; Flink can still place them in the right window.
- State is large or complex: each key keeps a sliding histogram, a sorted set of recent events, or a model score updated incrementally.
- You want exactly-once end to end with Kafka as both source and sink.
For simpler per-message enrichment with no cross-message state, a Kafka Streams app is faster to run. For ad hoc analysis over historical data, use Spark or a warehouse.
Event time vs processing time
Processing time
- Simple — no watermarks
- Low latency
- Results change if the pipeline lags
- Reprocessing gives different answers
Pick it when: Dashboards where approximate freshness is fine.
Event time
- Results depend only on the events, not when they arrived
- Reprocessing is deterministic
- Requires watermarks
- Windows close later
Pick it when: Fraud, billing, SLA metrics — anything where accuracy matters.
Flink timestamps every event using a field you choose, such as event_ts. It then advances a watermark — a lower bound on event time — to decide when a window is complete enough to emit results.
Watermarks
- Each event carries an event timestamp. Flink emits a watermark slightly behind the maximum observed timestamp, with a max-out-of-orderness gap such as 5 seconds.
- When the watermark crosses a window's end time, the window is considered complete and fires.
- Events arriving after the watermark — late data — can be handled with an allowed lateness period, producing a corrected result, or routed to a side output for separate handling.
Windows
Tumbling window
- Non-overlapping buckets
- Simple to reason about
- An event at the boundary falls in one window only
Pick it when: Minute-by-minute or hour-by-hour aggregations.
Sliding window
- Overlapping — every event appears in multiple windows
- Higher memory and CPU — N windows per event where N = window/slide
Pick it when: Rolling averages over the last 5 minutes, updated every minute.
Session window
- Closes when a user is idle for a gap duration
- Window size is data-driven, harder to plan capacity for
Pick it when: User sessions, click streams, IoT activity bursts.
All three can be keyed — each user, device or merchant gets its own independent window — which is how Flink scales: the key space fans out across parallel tasks.
Keyed state
Flink partitions the stream by a key using .keyBy(event -> event.userId). All events with the same key go to the same task, which maintains private state: value state for a counter, list state for recent events, map state for a histogram, reducing state for a running sum. State lives in a state backend: in-memory (fastest, limited size) or RocksDB (disk-spilling, terabytes per task).
- End-to-end latencywith processing-time or tight watermarks
- ~10–100 ms
- Throughput per nodedepends on state access and serialization
- millions of events/s
- State sizewith RocksDB backend
- terabytes per cluster
- Checkpoint intervalshorter increases overhead
- 30 s typical
Checkpoints and exactly-once
Flink takes consistent checkpoints using Chandy-Lamport snapshots. It injects a barrier into each Kafka partition. When a task receives barriers from all inputs, it writes its state to durable storage (S3 or HDFS) and forwards the barrier downstream. On failure, the entire job restarts from the last completed checkpoint, re-reading events from the Kafka offsets stored in that checkpoint.
- The job manager triggers a checkpoint with a monotonically increasing id.
- Barriers flow through the operator graph, carrying the checkpoint id.
- Each operator writes its state snapshot and acknowledges when all inputs have been aligned.
- The job manager marks the checkpoint complete. The Kafka source commits its consumer offsets at the same point.
- On failure, restart from the checkpoint, rewind the Kafka consumer to the stored offsets, and reprocess. Downstream Kafka sinks use idempotent producers or transactions to avoid duplicate output.
Backpressure
If a downstream task is slower than upstream, Flink uses TCP flow control through its network stack. Upstream tasks block when their output buffers fill, slowing the whole pipeline in a controlled way rather than dropping events. The Flink UI shows a backpressure ratio per task; when it's high, look for slow UDFs, state access bottlenecks, or an undersized state backend.
Where it bites
- Operational complexity. Flink needs a cluster (Flink on YARN, Kubernetes, or managed offerings such as Amazon Kinesis Data Analytics or Confluent Cloud for Flink). Kafka Streams runs as a library inside your application with no separate cluster.
- Savepoints for upgrades. Changing operator state schema or topology requires a savepoint and a coordinated restart. Plan this into your deployment process.
- Late data surprises. A spike of late mobile events can make a window emit a correction. Downstream consumers must handle updates to already-emitted results, or the job must use accumulating-and-retracting mode.
Flink vs the alternatives
Flink
- True event time
- Large keyed state
- Exactly-once end to end
- Low latency
- Separate cluster to operate
- Steep learning curve
Pick it when: Fraud, billing, metrics with exactly-once and event-time requirements.
Kafka Streams
- Library — no extra cluster
- Simple deployment
- JVM only
- Limited state per instance
- No built-in event-time watermarks
Pick it when: Per-record enrichment or simple aggregations within a Kafka-native stack.
Spark Structured Streaming
- Unified batch and streaming API
- Rich ecosystem
- Higher latency (micro-batch)
- Weaker event-time semantics
Pick it when: Teams that already use Spark for batch and want streaming on the same platform.
Common interview uses
- Fraud detection: key by card number, maintain a sliding window of transaction amounts, emit an alert when the windowed sum exceeds a threshold.
- Top-K items: key by item, increment a counter in a tumbling window, aggregate results with a global merge operator.
- Metrics rollups: consume an event stream, key by service and host, emit per-minute P99 latency values into a time-series store.
Try it
Event-time Stream Aggregation HLD
Design a pipeline that aggregates events by event time with correct handling of late arrivals.