Data Pipeline Skill

Building and reviewing ETL/ELT pipelines - an Agent Skill for Python 3.14, pandas 2.2+, and Polars 1.x.

What This Skill Does

Scaffolds batch pipelines: source connectors, validation layer, transform functions, partitioned outputs (Parquet/Delta), orchestration hooks (Airflow/dbt/Cron), and data-quality checks with reproducible run IDs.

When to Invoke

• New nightly ingest from SaaS API or warehouse export
• PR review on pandas Polars transform logic
• Adding validation after production null spike incident
• Migrating notebook prototype to scheduled job

Inputs

Outputs

• Pipeline module layout (extract.py, transform.py, load.py, validate.py)
• pyproject.toml deps pinned with uv
• Pandera or Great Expectations check stubs
• Partition path convention s3://bucket/dataset/dt=YYYY-MM-DD/
• Idempotent load strategy (merge keys, overwrite partition)
• Verification commands and row-count assertions

Guardrails

1. No silent dtype coercion - explicit schemas on read and write.
2. Validate before transform - fail fast on contract breaks.
3. Idempotent loads - reruns must not duplicate fact rows.
4. Secrets from env - never hard-code API tokens in notebooks committed to git.
5. Log run_id on every stage - correlate extract/transform/load logs.
6. Do not pandas-in-production without tests - minimum pytest on transform pure functions.

Recipe

uv run python -m pipeline.run --date 2026-07-08 --env staging
uv run pytest tests/test_transform.py -q
uv run python -m pipeline.validate --date 2026-07-08

Working Example (validate stage)

import pandera.pandas as pa
from pandera.typing import Series
 
 
class OrdersSchema(pa.DataFrameModel):
    order_id: Series[str] = pa.Field(unique=True)
    amount: Series[float] = pa.Field(ge=0)
    status: Series[str] = pa.Field(isin=["pending", "paid", "refunded"])
 
 
def validate_orders(df):
    return OrdersSchema.validate(df, lazy=True)

FAQs

Related

• Data Engineering Basics - patterns
• Data Validation and Quality - checks
• Airflow - orchestration
• pandas Basics - transform reference

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.