Data Engineering Highlights Summary

Every highlight bullet from the 11 pages in this section, gathered on one page and grouped by the page it came from.

The Batch, Streaming, and Orchestration Mental Model

• Batch trades latency for completeness; streaming trades completeness for latency
• A DAG encodes dependencies between tasks, not the literal order tasks run in
• Idempotency is what makes a retry, a backfill, or an at-least-once delivery guarantee safe
• Most delivery guarantees in practice are at-least-once - exactly-once is achieved by idempotent sinks, not magic

Data Engineering Basics

• Batch jobs process bounded datasets on a schedule
• Streaming handles unbounded events with low latency
• Idempotent tasks survive retries without duplicate facts
• Parquet partitions make pipelines replayable

Airflow

• DAGs define task dependencies - not execution order alone
• Tasks should be idempotent for retry safety
• catchup=False prevents historical stampede on deploy
• Use data intervals for partition keys, not wall clock alone

Prefect & Dagster

• Prefect flows are plain Python with @flow/@task
• Dagster models assets, not just task graphs
• Both support local dev then remote agents
• Pass paths and metadata - not giant DataFrames

dbt with Python

• dbt models are SELECT statements materialized in the warehouse
• Python models run where the adapter supports them
• tests and docs live beside SQL in git
• staging -> intermediate -> mart is the common layering

PySpark

• Spark distributes partitions across executors
• Avoid UDFs - prefer Spark SQL expressions
• shuffle is expensive - know your join keys
• write partitioned Parquet for downstream readers

Streaming (Kafka / Faust)

• Kafka is durable log; consumers track offsets
• Faust streams are async Kafka consumers with tables
• At-least-once is default - design idempotent sinks
• Windowed aggregations need event-time watermarks

Data Validation & Quality

• Validate at pipeline boundaries - not only in notebooks
• Pandera fits pandas/Polars in pytest and tasks
• Great Expectations suites document batch contracts
• Failed checks should block promotion, not warn silently

File Formats

• Parquet is columnar with embedded schema
• Arrow is in-memory columnar layout for zero-copy
• Partition directories act as hive keys
• zstd compression balances size and CPU

Workflow Reliability

• Retries without idempotency duplicate data
• Partition overwrite scopes safe reruns
• Alert on consumer lag and row-count anomalies
• Dead letter queues isolate poison messages

Data Engineering Best Practices

• Immutable raw layers - never edit in place
• Idempotent partition writes for safe retries
• Validate before promote, not after dashboards break
• Pin Python deps with uv lockfiles in CI