Data lakes, warehouses and Parquet

Organizations collect vast amounts of telemetry, transaction logs, and customer interactions. Storing and querying this data requires an analytical data architecture that balances cheap long-term retention against high-throughput analytical query performance.

Modern data platforms organize analytical storage around data lakehouses using standardized columnar formats like Apache Parquet.

Data lake vs data warehouse vs lakehouse

Row-oriented vs columnar storage

Relational databases (PostgreSQL, MySQL) store data in rows: all fields for row 1 are stored together on disk, followed by all fields for row 2. This is optimal for OLTP (e.g. SELECT * FROM users WHERE id = 42).

Analytical queries (OLAP), however, typically aggregate across billions of rows but touch only a few columns:

1 SELECT country, AVG(amount) FROM transactions WHERE date >= '2026-01-01' GROUP BY country;

In row-oriented storage, the engine must scan the entire 1 TB table from disk just to extract two columns. In columnar storage, each column is stored in separate contiguous disk blocks: the query engine reads only the country and amount blocks, reducing disk IO by 90% or more.

Inside Apache Parquet

Apache Parquet is the open-source standard columnar file format for analytical systems:

Hive partitioning and partition pruning

Data lakes organize Parquet files in object storage using hierarchical directory paths:

1 s3://analytics-bucket/events/year=2026/month=10/day=17/part-001.parquet

When a user executes a query with WHERE year = 2026 AND month = 10 AND day = 17, the query engine applies partition pruning: it reads only the specific day directory, skipping petabytes of historical files completely.