File Formats

Columnar formats (Parquet, Arrow) dominate analytics pipelines: compress well, embed schema, and let readers load only needed columns.

Recipe

Quick-reference recipe card - copy-paste ready.

import pandas as pd
 
df = pd.read_csv("orders.csv", parse_dates=["ordered_at"])
df.to_parquet(
    "lake/orders/dt=2025-01-15/data.parquet",
    index=False,
    compression="zstd",
)
# Read back selected columns
subset = pd.read_parquet("lake/orders", columns=["order_id", "revenue"], filters=[("dt", ">=", "2025-01-01")]

When to reach for this:

• Lakehouse raw and mart layers on object storage
• Handoff between Python, Spark, Polars, and DuckDB
• Reducing storage cost vs gzip CSV
• Schema evolution with documented columns

Working Example

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from pathlib import Path
 
df = pd.DataFrame(
    {
        "order_id": [1, 2, 3],
        "region": pd.Categorical(["East", "West", "East"]),
        "revenue": [120.0, 340.0, 150.0],
        "ordered_at": pd.to_datetime(["2025-01-01", "2025-01-02", "2025-01-03"], utc=True),
    }
)
 
# pandas -> Arrow Table (zero-copy where possible)
table = pa.Table.from_pandas(df, preserve_index=False)
 
out_dir = Path("lake/orders/dt=2025-01-15")
out_dir.mkdir(parents=True, exist_ok=True)
pq.write_table(
    table,
    out_dir / "data.parquet",
    compression="zstd",
    use_dictionary=True,
)
 
# Read metadata without loading data
meta = pq.read_metadata(out_dir / "data.parquet")
print("rows:", meta.num_rows, "columns:", meta.num_columns)
 
# Column pruning read
loaded = pq.read_table(out_dir / "data.parquet", columns=["order_id", "revenue"])
print(loaded.to_pandas())

What this demonstrates:

• Hive partition directory dt=YYYY-MM-DD
• Arrow as interchange between pandas and Parquet writer
• Metadata inspection for ops checks
• Column pruning on read

Deep Dive

How It Works

• Parquet stores column chunks with statistics (min/max) per row group.
• Arrow defines in-memory layout; pandas 2.2+ uses Arrow-backed strings.
• Readers skip row groups using statistics when filters match.
• Partition folders (key=value/) emulate warehouse partitioning on files.

Format Comparison

Python Notes

import polars as pl
 
# Polars scan with projection pushdown
lf = pl.scan_parquet("lake/orders/**/*.parquet").select("order_id", "revenue")

Gotchas

• Writing index=True - surprise index column in Parquet. Fix: index=False always in pipelines.
• Small files problem - thousands of 1 MB files slow listing. Fix: target 128-512 MB per file; compact partitions.
• Schema drift across days - mergeSchema hides breaking changes. Fix: explicit schema registry and validation.
• Timezone-naive timestamps in Parquet - ambiguous replays. Fix: store UTC with dtype datetime64[ns, UTC].
• CSV as mart format - re-parse cost every read. Fix: Parquet marts; CSV only for human export.

Alternatives

FAQs

import duckdb
duckdb.sql("SELECT region, SUM(revenue) FROM read_parquet('lake/orders/**') GROUP BY 1")

Related

• Performance & Memory - column pruning
• PySpark - distributed Parquet writes
• Data Engineering Basics - partition overwrite
• Data Validation & Quality - schema gates

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.