LESSON 30.1
Distributed compute with MapReduce and Spark
Scaling data processing across clusters with map shuffle reduce primitives, Apache Spark resilient distributed datasets, in-memory pipelining, and fault recovery.
6 min read
When data sets expand beyond a single server's disk and memory capacity (hundreds of gigabytes to petabytes), queries cannot run on standard relational databases. Processing large volumes of raw event logs, clickstreams, or image collections requires distributing computation across a cluster of hundreds or thousands of worker machines.
The foundation of modern big data processing is distributed batch compute, evolving from Google's original MapReduce paper to modern Apache Spark.
The MapReduce paradigm
MapReduce breaks complex analytical queries down into three sequential steps:
- Map: Workers read input splits in parallel from distributed storage (like HDFS or S3). Each mapper processes records independently, transforming inputs into intermediate key-value pairs (
key, value). - Shuffle and Sort: The framework partitions intermediate keys (typically by hashing
hash(key) % num_reducers) and transfers matching keys across the network to designated reducer nodes. - Reduce: Reducer workers iterate over all values associated with each unique key, computing aggregations (sums, counts, averages, joins) and writing final outputs back to distributed storage.
The shuffle bottleneck
The shuffle phase is the most dangerous performance bottleneck in distributed compute:
If intermediate data has high skew (for example, 90% of events share a single popular key), a single reducer receives 90% of the shuffle traffic, stalling the entire job while other reducers sit idle.
Why Spark replaced MapReduce: In-memory RDDs
In Hadoop MapReduce, every multi-stage workflow forces each job stage to write its output to replicated physical disk blocks before the next job stage can read it. A 5-stage data transformation incurs five rounds of disk serialization and network replication.
Apache Spark introduced Resilient Distributed Datasets (RDDs):
Hadoop MapReduce
- Extremely resilient to mid-job node crashes on low-spec hardware
- Low RAM requirements
- Mandatory disk spills after every map and reduce phase
- 10x to 100x slower on iterative machine learning pipelines
Pick it when: Legacy petabyte batch jobs on cheap commodity hardware.
Apache Spark
- Pipelines transformations in memory without intermediate disk writes
- Rich APIs (Spark SQL, DataFrames, GraphX)
- Up to 100x faster than MapReduce in RAM
- High RAM footprint
- JVM garbage collection pauses on large executor heaps
Pick it when: Modern batch ETL pipelines, interactive data analytics, feature engineering.
Fault tolerance through lineage graphs
Rather than replicating data partitions across machines in RAM, Spark achieves fault tolerance through DAG lineage graphs:
- Spark tracks the exact sequence of deterministic transformations (
filter,map,groupByKey) used to build each dataset partition. - If Worker Node 4 crashes and loses Partition 12 from RAM, Spark does not restart the entire cluster job.
- The scheduler simply consults the lineage graph and re-computes Partition 12 on an available worker by replaying the specific transformations on the original source block.
Try it
Multi-tenant Analytics Query Platform HLD
Design a scalable analytical query platform capable of executing complex aggregations over billions of records.