Data Validation & Quality

Schema and statistical checks at pipeline boundaries catch bad ingests before they pollute marts - Pandera for Python DataFrames, Great Expectations for documented data contracts.

Recipe

Quick-reference recipe card - copy-paste ready.

import pandera.pandas as pa
 
schema = pa.DataFrameSchema({
    "order_id": pa.Column(int, unique=True, nullable=False),
    "revenue": pa.Column(float, pa.Check.ge(0)),
    "region": pa.Column(str, pa.Check.isin(["East", "West", "Central"])),
})
 
def validate_orders(df):
    return schema.validate(df, lazy=True)

When to reach for this:

• Every ETL stage before overwrite writes
• CI tests on fixture Parquet samples
• SLA dashboards on freshness and volume
• Regulatory or finance data needing audit trail

Working Example

import pandas as pd
import pandera.pandas as pa
from pandera import Check, Column, DataFrameSchema
 
orders_schema = DataFrameSchema(
    {
        "order_id": Column(int, unique=True, nullable=False),
        "revenue": Column(float, checks=Check.ge(0)),
        "region": Column(str, checks=Check.isin(["East", "West", "Central"])),
        "ordered_at": Column("datetime64[ns, UTC]", nullable=False),
    },
    strict=False,
    coerce=True,
)
 
def load_orders(path: str) -> pd.DataFrame:
    df = pd.read_parquet(path)
    try:
        return orders_schema.validate(df, lazy=True)
    except pa.errors.SchemaErrors as err:
        # Quarantine bad rows for inspection
        failure_cases = err.failure_cases
        failure_cases.to_csv("quarantine/schema_failures.csv", index=False)
        raise
 
# Great Expectations style check (conceptual GE API)
def gx_style_checks(df: pd.DataFrame) -> None:
    assert len(df) > 0, "empty batch"
    assert df["order_id"].is_unique.all()
    assert df["revenue"].between(0, 1_000_000).mean() > 0.99

What this demonstrates:

• Pandera schema with coercion and lazy multi-error report
• Quarantine artifact on failure
• Lightweight assertions complementing formal schemas
• UTC datetime column typing

Deep Dive

How It Works

• Pandera validates DataFrame shape, dtypes, and custom checks in Python.
• Great Expectations stores Expectation Suites, validates batches, renders Data Docs.
• Fail closed: orchestrator task should not promote mart on validation error.
• Combine schema (structure) with anomaly checks (volume, null rate).

Check Types

Python Notes

# Polars with pandera.polars
import polars as pl
import pandera.polars as pa
 
schema = pa.DataFrameSchema({"id": pa.Column(int)})
schema.validate(pl.DataFrame({"id": [1, 2]}))

Gotchas

• Validating only in notebook - production bypasses checks. Fix: call validator in @task before write.
• lazy=False on wide schemas - stops at first error. Fix: lazy=True for full failure report.
• Brittle exact row counts - flaky on real traffic. Fix: range checks vs trailing average.
• Warning-only failures - bad data still ships. Fix: raise and block downstream sensors.
• Schema drift unversioned - deploy breaks silently. Fix: git tag schema versions with pipeline code.

Y29kZWd1aWRlcy5pb3xjZ2lvNzM4fDIwMjYwNw==

Alternatives

FAQs

import pytest
import pandera.errors
 
def test_rejects_negative_revenue(bad_df):
    with pytest.raises(pandera.errors.SchemaError):
        orders_schema.validate(bad_df)

Related

• Cleaning & Transforming Data - fixes before validate
• dbt with Python - warehouse tests
• Data Engineering Best Practices - quality gates
• Workflow Reliability - fail vs retry

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.