Warehouse transforms with tests/docs. Python ETL lands clean staging tables dbt refs.

Sample 10k-row Parquet slices committed or generated in CI fixtures. Never copy prod PII to laptops.

pandas/Polars on single node until measured pain. Spark when data and join shuffle exceed one machine.

Add dbt build or Pandera validation step when applicable.

Data Engineering Best Practices

Q: How do I version marts?

Breaking schema: new table fct_orders_v2 or version column with migration window.

Q: How do I handle late-arriving facts?

Document merge window (e.g., reprocess last 3 dt partitions nightly).

Testable, observable, reproducible pipelines that survive retries, schema drift, and on-call nights.

How to Use This List

Apply sections A-D when designing a new DAG, flow, or asset.
Use as PR checklist for any change touching ingest, transforms, or marts.
Pair with Workflow Reliability when incidents occur.

A - Data Lake & Storage

Raw layer is append-only. New data lands in dated partitions; never mutate historical raw files.
Parquet with zstd for analytics layers. Embed schema; avoid CSV marts.
Partition by logical batch key. dt= or run_id= enables idempotent overwrite.
Target right-sized files. Compact small shards to ~128-512 MB objects.
Store timestamps in UTC. Document display timezone separately.

B - Code & Testing

Pure functions for transform logic. Test with pytest and tmp_path Parquet without orchestrator.
Pin dependencies with uv lockfiles. pandas 2.2+, Polars 1.x, pyarrow versions match CI and prod.
Schema tests on keys and ranges. Pandera or dbt tests block bad batches.
No heavy work at DAG parse time. Imports and I/O only inside tasks.
Pass URIs not DataFrames in orchestrator state. XCom/metadata store is not a data lake.

C - Operations & Reliability

Log run_id, partition, input_rows, output_rows, duration. Structured JSON for search.
Retries only with idempotency. Partition overwrite or merge keys - see workflow guide.
Alert on final failure and SLA miss. Not every transient retry.
DLQ or quarantine for poison events. Schema failures export row-level diagnostics securely.
Document backfill procedure. Include max_active_runs and validation gates.

D - Governance & Handoff

Layer models: staging, intermediate, mart. Consumers read marts only.
PII tagged and minimized early. Hash in staging; restrict quarantine access.
Lineage in git. dbt manifest, DAG repo, or Dagster asset graph - one source of truth.
Data contracts with producers. Volume and schema change notices before deploy.
Postmortems update this checklist. Add rules when incidents reveal gaps.

FAQs

Batch or stream first?

Start batch daily/hourly until latency requirements force streaming.
Micro-batch often covers "near real-time" cheaper.

Airflow, Prefect, or Dagster?

Use what ops can run reliably today.
Greenfield Python teams often pick Prefect/Dagster; enterprises may already have Airflow.

Where does dbt fit?

Warehouse transforms with tests/docs.
Python ETL lands clean staging tables dbt refs.

How much testing is enough?

Schema tests on every primary key column.
Unit tests on non-trivial transform functions.
One integration test on sample partition end-to-end.

Local dev data?

Sample 10k-row Parquet slices committed or generated in CI fixtures.
Never copy prod PII to laptops.

How do I version marts?

Breaking schema: new table fct_orders_v2 or version column with migration window.

Spark vs pandas?

pandas/Polars on single node until measured pain.
Spark when data and join shuffle exceed one machine.

How do I handle late-arriving facts?

Document merge window (e.g., reprocess last 3 dt partitions nightly).

CI pipeline?

uv sync && pytest && ruff check . && python -m jobs.daily_orders --dry-run

Add dbt build or Pandera validation step when applicable.

Who owns data quality?

Producers own schema; pipeline owners enforce gates; consumers flag anomalies.

Data Engineering Basics - lifecycle intro
Workflow Reliability - idempotency workflow
Data Validation & Quality - schema gates
File Formats - Parquet layout
Data Analysis Best Practices - upstream analysis habits

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Data Engineering Best Practices

Testable, observable, reproducible pipelines that survive retries, schema drift, and on-call nights.

How to Use This List

Apply sections A-D when designing a new DAG, flow, or asset.
Use as PR checklist for any change touching ingest, transforms, or marts.
Pair with Workflow Reliability when incidents occur.

A - Data Lake & Storage

Raw layer is append-only. New data lands in dated partitions; never mutate historical raw files.
Parquet with zstd for analytics layers. Embed schema; avoid CSV marts.
Partition by logical batch key. dt= or run_id= enables idempotent overwrite.
Target right-sized files. Compact small shards to ~128-512 MB objects.
Store timestamps in UTC. Document display timezone separately.

B - Code & Testing

Pure functions for transform logic. Test with pytest and tmp_path Parquet without orchestrator.
Pin dependencies with uv lockfiles. pandas 2.2+, Polars 1.x, pyarrow versions match CI and prod.
Schema tests on keys and ranges. Pandera or dbt tests block bad batches.
No heavy work at DAG parse time. Imports and I/O only inside tasks.
Pass URIs not DataFrames in orchestrator state. XCom/metadata store is not a data lake.

C - Operations & Reliability

Log run_id, partition, input_rows, output_rows, duration. Structured JSON for search.
Retries only with idempotency. Partition overwrite or merge keys - see workflow guide.
Alert on final failure and SLA miss. Not every transient retry.
DLQ or quarantine for poison events. Schema failures export row-level diagnostics securely.
Document backfill procedure. Include max_active_runs and validation gates.

D - Governance & Handoff

Layer models: staging, intermediate, mart. Consumers read marts only.
PII tagged and minimized early. Hash in staging; restrict quarantine access.
Lineage in git. dbt manifest, DAG repo, or Dagster asset graph - one source of truth.
Data contracts with producers. Volume and schema change notices before deploy.
Postmortems update this checklist. Add rules when incidents reveal gaps.

FAQs

Batch or stream first?

Start batch daily/hourly until latency requirements force streaming.
Micro-batch often covers "near real-time" cheaper.

Airflow, Prefect, or Dagster?

Use what ops can run reliably today.
Greenfield Python teams often pick Prefect/Dagster; enterprises may already have Airflow.

Where does dbt fit?

Warehouse transforms with tests/docs.
Python ETL lands clean staging tables dbt refs.

How much testing is enough?

Schema tests on every primary key column.
Unit tests on non-trivial transform functions.
One integration test on sample partition end-to-end.

Local dev data?

Sample 10k-row Parquet slices committed or generated in CI fixtures.
Never copy prod PII to laptops.

How do I version marts?

Breaking schema: new table fct_orders_v2 or version column with migration window.

Spark vs pandas?

pandas/Polars on single node until measured pain.
Spark when data and join shuffle exceed one machine.

How do I handle late-arriving facts?

Document merge window (e.g., reprocess last 3 dt partitions nightly).

CI pipeline?

uv sync && pytest && ruff check . && python -m jobs.daily_orders --dry-run

Add dbt build or Pandera validation step when applicable.

Who owns data quality?

Producers own schema; pipeline owners enforce gates; consumers flag anomalies.

Data Engineering Basics - lifecycle intro
Workflow Reliability - idempotency workflow
Data Validation & Quality - schema gates
File Formats - Parquet layout
Data Analysis Best Practices - upstream analysis habits

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Data Engineering Best Practices

Search Documentation

Data Engineering Best Practices