Data Engineering Basics

9 examples to get you started with Data Engineering - 7 basic and 2 intermediate.

Prerequisites

• Python 3.14.0 with pandas 2.2+, pyarrow, and orchestration familiarity.
• Quick setup: uv pip install pandas pyarrow prefect

Basic Examples

1. Extract to Parquet

Land raw data in columnar storage before transforms.

import pandas as pd
 
df = pd.read_csv("orders.csv", parse_dates=["ordered_at"], dtype={"region": "category"})
df.to_parquet("raw/orders.parquet", index=False, compression="zstd")

• Parquet embeds schema and compresses better than CSV.
• Partition paths like dt=2025-01-15/ speed selective reads.
• Never overwrite raw - append partitions by run date.

Related: File Formats - Parquet and Arrow

2. Transform with pandas

Clean and shape data in a scriptable step.

import pandas as pd
 
df = pd.read_parquet("raw/orders.parquet")
clean = (
    df.dropna(subset=["order_id"])
    .assign(revenue=pd.to_numeric(df["revenue"], errors="coerce"))
    .query("revenue >= 0")
)
clean.to_parquet("staging/orders_clean.parquet", index=False)

• Each stage writes an artifact you can audit.
• Log input/output row counts every step.
• Keep business rules in version-controlled Python.

Related: Cleaning & Transforming Data

3. Idempotent Load

Upsert or partition overwrite so retries do not duplicate.

import pandas as pd
from pathlib import Path
 
run_dt = "2025-01-15"
out = Path(f"mart/orders/dt={run_dt}/data.parquet")
out.parent.mkdir(parents=True, exist_ok=True)
df.to_parquet(out, index=False)  # overwrite partition for this dt

• Partition key scopes idempotency to one day.
• Re-running the same run_dt replaces, not appends duplicates.
• Document late-arriving data policy separately.

Related: Workflow Reliability - idempotency decisions

4. Schedule with cron

Batch jobs run on a clock when latency tolerance is hours.

# crontab: 0 6 * * * /path/.venv/bin/python /path/jobs/daily_orders.py
import logging
logging.basicConfig(level=logging.INFO)
logging.info("daily_orders started")

• Cron is fine for single-server MVP pipelines.
• Redirect logs to files or centralized logging.
• Exit non-zero on failure so monitoring alerts.

Related: Airflow - DAG scheduling

5. Task Boundaries

One function per step with typed inputs/outputs.

from pathlib import Path
import pandas as pd
 
def extract(path: Path) -> pd.DataFrame:
    return pd.read_parquet(path)
 
def transform(df: pd.DataFrame) -> pd.DataFrame:
    return df.groupby("region", observed=True)["revenue"].sum().reset_index()
 
def load(df: pd.DataFrame, path: Path) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    df.to_parquet(path, index=False)

• Pure functions simplify unit tests.
• Pass paths or configs - avoid hidden globals.
• Orchestrator wires extract -> transform -> load.

Related: Prefect & Dagster

6. Validate Schema

Reject bad data before it reaches marts.

import pandera.pandas as pa
 
schema = pa.DataFrameSchema({
    "order_id": pa.Column(int, unique=True),
    "revenue": pa.Column(float, pa.Check.ge(0)),
})
schema.validate(df)

• Validation failures should block downstream tasks.
• Store quarantine Parquet for rejected rows.
• Version schemas alongside pipeline code.

Related: Data Validation & Quality

7. Log Row Counts

Operational visibility starts with simple metrics.

import logging
logger = logging.getLogger(__name__)
 
def transform(df):
    logger.info("input_rows=%s", len(df))
    out = df.drop_duplicates("order_id")
    logger.info("output_rows=%s", len(out))
    return out

• Log duration, row counts, and partition keys.
• Compare to historical ranges for anomaly detection.
• Structured JSON logs parse better in Loki/ELK.

Related: Data Engineering Best Practices

Intermediate Examples

8. Prefect Flow Decorator

Orchestrate Python tasks with retries and UI.

from prefect import flow, task
import pandas as pd
 
@task(retries=2, retry_delay_seconds=30)
def extract() -> pd.DataFrame:
    return pd.read_parquet("raw/orders.parquet")
 
@flow(log_prints=True)
def daily_orders():
    df = extract()
    # transform + load ...
 
if __name__ == "__main__":
    daily_orders()

• @task retries isolate transient S3/DB failures.
• Prefect 2.x flows are plain Python - no XML DAG files.
• Run locally first; agent deploys to production.

Related: Prefect & Dagster

9. Read Partitioned Dataset

Scan only the dates you need.

import pandas as pd
 
df = pd.read_parquet("mart/orders", filters=[("dt", ">=", "2025-01-01")]

• Hive-style partitions reduce I/O on historical replays.
• Polars scan_parquet offers the same predicate pushdown.
• Keep partition column in path, not duplicated in every row unless needed.

Related: File Formats - partition layout

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.