LESSON 30.4
Orchestration, backfills and data quality
Pipelines as DAGs in tools like Airflow and Dagster, idempotent partition overwrites, backfilling history safely, and data quality checks and contracts that stop bad data spreading.
6 min read
A data platform is rarely one job. It's hundreds of jobs that depend on each other: ingest raw events, clean them, join with customers, aggregate daily, feed dashboards and models. Running them reliably is a discipline of its own.
Pipelines as DAGs
Orchestrators like Airflow, Dagster and Prefect describe a pipeline as a directed acyclic graph of tasks. Each task runs when its upstream tasks have succeeded for the same period.
The orchestrator schedules runs, retries failed tasks, tracks which periods succeeded and shows lineage: if "Join" failed for October 17, it knows "Daily revenue" and "Dashboard" are stale for that day.
Idempotent, partitioned tasks
Retries and re-runs are routine, so every task must be safe to run again. The standard pattern:
- Each run owns a partition, usually a date: the run for October 17 reads that day's inputs and writes
date=2026-10-17. - Overwrite, don't append. The task replaces its output partition entirely, written to a temporary location and swapped in atomically.
- Inputs are fixed. A run reads data for its own period, not "whatever is new since last time", so re-running gives the same result.
With this, a retry after a crash, or a deliberate re-run, never double counts.
Backfills
When logic changes or a bug is found, history must be recomputed: a backfill. Partitioned, idempotent tasks make it mechanical: re-run the task for each affected date, then re-run everything downstream. Practical concerns:
Run backfill alongside daily jobs
- History fixed quickly
- Competes for cluster and warehouse capacity
Pick it when: Short ranges, spare capacity.
Throttled or separate pool
- Daily runs stay on time
- Backfill takes longer
Pick it when: Long ranges like months or years.
Tell consumers a backfill is happening: dashboards and models may briefly mix old and corrected numbers.
Data quality
The worst pipeline failures don't crash. They publish wrong data: nulls after an upstream rename, duplicated rows after a retry bug, half a day's events after a late file. Add checks at each stage:
- Row count
- within the usual range compared with the last few weeks
- Not-null and unique
- keys present and unique
- Value ranges
- no negative prices, dates in the expected period
- Freshness
- the latest data arrived on time
- Referential
- every order's customer exists
Failed checks should block publication (a "write-audit-publish" pattern: write to staging, check, then swap in), and alert the owner. Tools like dbt tests and Great Expectations make these declarative.
Try it
Large Data Export Platform HLD
Design a recurring export that is safe to retry and backfill.