LESSON 30.3
Batch vs streaming, lambda and kappa
Architectural evolution of data pipelines. Reconciling high-latency accurate batch runs with low-latency streaming approximations using Lambda and Kappa architectures.
6 min read
Data pipelines have two competing requirements: speed and accuracy. Business dashboards and fraud detection engines demand fresh numbers updated in sub-second real time. At the same time, financial audits, monthly billing, and ML models require 100% complete, deduplicated, and globally consistent historical truth.
Reconciling these conflicting goals drove the evolution from Batch processing to the Lambda and Kappa architectures.
The trade-off: Latency vs completeness
Batch Processing (Hourly / Daily)
- Computes over complete, bounded datasets
- Tolerates late-arriving data cleanly
- Cost-effective, highly optimized bulk throughput
- High data latency (hours or days before insights appear)
Pick it when: Daily billing reconciliation, machine learning model training, compliance reporting.
Stream Processing (Real-time)
- Processes unbounded event streams with sub-second latency
- Immediate alerts and dynamic real-time dashboards
- Must handle out-of-order and duplicate events
- Higher state management complexity in memory
Pick it when: Fraud detection, real-time metrics, live user recommendations.
The Lambda architecture
To achieve both real-time speed and historical accuracy, Nathan Marz proposed the Lambda Architecture:
- Batch Layer: Immutable append-only raw data stored in S3/HDFS. A daily batch job runs over all historical data, computing 100% accurate pre-aggregated views.
- Speed Layer: A real-time streaming engine (Flink, Storm) processes events arriving since the last batch run, updating an ephemeral low-latency delta view.
- Serving Layer: Analytical queries combine both views:
Result = BatchView + SpeedView. As the batch job finishes each day, the new batch view supersedes and replaces the temporary speed layer results.
The fatal flaw of Lambda: Engineers had to write, debug, and maintain every single transformation algorithm twice—once in Java for Spark batch, and once in Scala/Java for the streaming engine.
The Kappa architecture
With the advent of modern distributed event logs (like Kafka) and unified stream-processing frameworks (like Apache Flink), Jay Kreps proposed the Kappa Architecture:
- Single processing engine: Eliminate the batch layer entirely. Everything is a stream.
- Long-retention event log: Retain raw immutable events in Kafka or tiered object storage for months or years.
- Streaming first: A single streaming pipeline processes incoming events continuously in real time.
- Historical reprocessing: When business logic changes or a bug is fixed, deploy a second instance of the streaming job configured with
offset = 0. The new job replays historical events from the beginning of the log at high speed into a new read table. Once caught up, switch queries to the new table and decommission the old job.
Event time vs processing time and watermarks
Streaming pipelines must deal with mobile devices going offline (in tunnels, airplane mode) and sending buffered events hours later:
- Processing time: The wall-clock time of the server processing the event. Simple, but produces inaccurate aggregations if network lags or outages occur.
- Event time: The timestamp when the user actually performed the action on their device.
- Watermarks: A streaming watermark is a timestamp metric indicating that the system assumes all events up to time $T$ have arrived. Any event arriving with a timestamp older than the watermark is classified as "late data" and either routed to a Dead Letter Queue or used to trigger a retroactive window recalculation.
Try it
Multi-tenant Analytics Query Platform HLD
Design an event analytics pipeline that handles late-arriving events using watermarks and streaming window aggregations.