File Formats

Columnar formats (Parquet, Arrow) dominate analytics pipelines: compress well, embed schema, and let readers load only needed columns.

Recipe

Quick-reference recipe card - copy-paste ready.

import pandas as pd
 
df = pd.read_csv("orders.csv", parse_dates=["ordered_at"])
df.to_parquet(
    "lake/orders/dt=2025-01-15/data.parquet",
    index=False,
    compression="zstd",
)
# Read back selected columns
subset = pd.read_parquet("lake/orders", columns=["order_id", "revenue"], filters=[("dt", ">=", "2025-01-01")]

When to reach for this:

Lakehouse raw and mart layers on object storage
Handoff between Python, Spark, Polars, and DuckDB
Reducing storage cost vs gzip CSV
Schema evolution with documented columns

Working Example

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from pathlib import Path
 
df = pd.DataFrame(
    {
        "order_id": [1, 2, 3],
        "region": pd.Categorical(["East", "West", "East"]),
        "revenue": [120.0, 340.0, 150.0],
        "ordered_at": pd.to_datetime(["2025-01-01", "2025-01-02", "2025-01-03"], utc=True),
    }
)
 
# pandas -> Arrow Table (zero-copy where possible)
table = pa.Table.from_pandas(df, preserve_index=False)
 
out_dir = Path("lake/orders/dt=2025-01-15")
out_dir.mkdir(parents=True, exist_ok=True)
pq.write_table(
    table,
    out_dir / "data.parquet",
    compression="zstd",
    use_dictionary=True,
)
 
# Read metadata without loading data
meta = pq.read_metadata(out_dir / "data.parquet")
print("rows:", meta.num_rows, "columns:", meta.num_columns)
 
# Column pruning read
loaded = pq.read_table(out_dir / "data.parquet", columns=["order_id", "revenue"])
print(loaded.to_pandas())

What this demonstrates:

Hive partition directory dt=YYYY-MM-DD
Arrow as interchange between pandas and Parquet writer
Metadata inspection for ops checks
Column pruning on read

Deep Dive

How It Works

Parquet stores column chunks with statistics (min/max) per row group.
Arrow defines in-memory layout; pandas 2.2+ uses Arrow-backed strings.
Readers skip row groups using statistics when filters match.
Partition folders (key=value/) emulate warehouse partitioning on files.

Format Comparison

Format	Strength	Weakness
CSV	Human readable	No schema, fat, slow
JSON lines	Semi-structured events	Verbose, slow analytics
Parquet	Analytics standard	Not human readable
Arrow IPC	Fast local IPC	Not long-term archive

Python Notes

import polars as pl
 
# Polars scan with projection pushdown
lf = pl.scan_parquet("lake/orders/**/*.parquet").select("order_id", "revenue")

Gotchas

Writing index=True - surprise index column in Parquet. Fix: index=False always in pipelines.
Small files problem - thousands of 1 MB files slow listing. Fix: target 128-512 MB per file; compact partitions.
Schema drift across days - mergeSchema hides breaking changes. Fix: explicit schema registry and validation.
Timezone-naive timestamps in Parquet - ambiguous replays. Fix: store UTC with dtype datetime64[ns, UTC].
CSV as mart format - re-parse cost every read. Fix: Parquet marts; CSV only for human export.

Alternatives

Alternative	Use When	Don't Use When
Delta/Iceberg/Hudi	ACID upserts on lake	Simple daily overwrite partitions
Avro	Kafka schema evolution	Analytics column pruning
ORC	Hive legacy clusters	Greenfield Python lake
SQLite DuckDB file	Local embedded analytics	Multi-writer object storage

FAQs

snappy or zstd?

zstd better compression ratio; slightly more CPU.
Default zstd for archives; snappy for ultra-low latency writes.

How big should Parquet files be?

Target 128-512 MB uncompressed equivalent per file.
Compaction jobs merge small daily shards monthly.

Partition column in data too?

Redundant if only in path - saves space.
Include if consumers query without partition pruning.

How do I read with DuckDB?

import duckdb
duckdb.sql("SELECT region, SUM(revenue) FROM read_parquet('lake/orders/**') GROUP BY 1")

Arrow vs Parquet?

Arrow: in-memory interchange.
Parquet: on-disk persistence format.

How do I encrypt at rest?

SSE on S3/GCS bucket policies.
Client-side encryption rare unless compliance mandates.

Nested data?

Parquet supports structs/lists for JSON-like events.
Flatten for BI-friendly marts when possible.

How do I diff schemas?

pyarrow.parquet.read_schema on two paths and compare.
Automate in CI on new partitions.

pandas vs pyarrow write?

pandas to_parquet uses pyarrow/fastparquet under hood.
pq.write_table when you already have Arrow Table.

Polars and pandas interchange?

Arrow bridge: pl.from_pandas / to_pandas zero-copy when dtypes align.

Performance & Memory - column pruning
PySpark - distributed Parquet writes
Data Engineering Basics - partition overwrite
Data Validation & Quality - schema gates

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

File Formats

Columnar formats (Parquet, Arrow) dominate analytics pipelines: compress well, embed schema, and let readers load only needed columns.

Recipe

Quick-reference recipe card - copy-paste ready.

import pandas as pd
 
df = pd.read_csv("orders.csv", parse_dates=["ordered_at"])
df.to_parquet(
    "lake/orders/dt=2025-01-15/data.parquet",
    index=False,
    compression="zstd",
)
# Read back selected columns
subset = pd.read_parquet("lake/orders", columns=["order_id", "revenue"], filters=[("dt", ">=", "2025-01-01")]

When to reach for this:

Lakehouse raw and mart layers on object storage
Handoff between Python, Spark, Polars, and DuckDB
Reducing storage cost vs gzip CSV
Schema evolution with documented columns

Working Example

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from pathlib import Path
 
df = pd.DataFrame(
    {
        "order_id": [1, 2, 3],
        "region": pd.Categorical(["East", "West", "East"]),
        "revenue": [120.0, 340.0, 150.0],
        "ordered_at": pd.to_datetime(["2025-01-01", "2025-01-02", "2025-01-03"], utc=True),
    }
)
 
# pandas -> Arrow Table (zero-copy where possible)
table = pa.Table.from_pandas(df, preserve_index=False)
 
out_dir = Path("lake/orders/dt=2025-01-15")
out_dir.mkdir(parents=True, exist_ok=True)
pq.write_table(
    table,
    out_dir / "data.parquet",
    compression="zstd",
    use_dictionary=True,
)
 
# Read metadata without loading data
meta = pq.read_metadata(out_dir / "data.parquet")
print("rows:", meta.num_rows, "columns:", meta.num_columns)
 
# Column pruning read
loaded = pq.read_table(out_dir / "data.parquet", columns=["order_id", "revenue"])
print(loaded.to_pandas())

What this demonstrates:

Hive partition directory dt=YYYY-MM-DD
Arrow as interchange between pandas and Parquet writer
Metadata inspection for ops checks
Column pruning on read

Deep Dive

How It Works

Parquet stores column chunks with statistics (min/max) per row group.
Arrow defines in-memory layout; pandas 2.2+ uses Arrow-backed strings.
Readers skip row groups using statistics when filters match.
Partition folders (key=value/) emulate warehouse partitioning on files.

Format Comparison

Format	Strength	Weakness
CSV	Human readable	No schema, fat, slow
JSON lines	Semi-structured events	Verbose, slow analytics
Parquet	Analytics standard	Not human readable
Arrow IPC	Fast local IPC	Not long-term archive

Python Notes

import polars as pl
 
# Polars scan with projection pushdown
lf = pl.scan_parquet("lake/orders/**/*.parquet").select("order_id", "revenue")

Gotchas

Writing index=True - surprise index column in Parquet. Fix: index=False always in pipelines.
Small files problem - thousands of 1 MB files slow listing. Fix: target 128-512 MB per file; compact partitions.
Schema drift across days - mergeSchema hides breaking changes. Fix: explicit schema registry and validation.
Timezone-naive timestamps in Parquet - ambiguous replays. Fix: store UTC with dtype datetime64[ns, UTC].
CSV as mart format - re-parse cost every read. Fix: Parquet marts; CSV only for human export.

Alternatives

Alternative	Use When	Don't Use When
Delta/Iceberg/Hudi	ACID upserts on lake	Simple daily overwrite partitions
Avro	Kafka schema evolution	Analytics column pruning
ORC	Hive legacy clusters	Greenfield Python lake
SQLite DuckDB file	Local embedded analytics	Multi-writer object storage

FAQs

snappy or zstd?

zstd better compression ratio; slightly more CPU.
Default zstd for archives; snappy for ultra-low latency writes.

How big should Parquet files be?

Target 128-512 MB uncompressed equivalent per file.
Compaction jobs merge small daily shards monthly.

Partition column in data too?

Redundant if only in path - saves space.
Include if consumers query without partition pruning.

How do I read with DuckDB?

import duckdb
duckdb.sql("SELECT region, SUM(revenue) FROM read_parquet('lake/orders/**') GROUP BY 1")

Arrow vs Parquet?

Arrow: in-memory interchange.
Parquet: on-disk persistence format.

How do I encrypt at rest?

SSE on S3/GCS bucket policies.
Client-side encryption rare unless compliance mandates.

Nested data?

Parquet supports structs/lists for JSON-like events.
Flatten for BI-friendly marts when possible.

How do I diff schemas?

pyarrow.parquet.read_schema on two paths and compare.
Automate in CI on new partitions.

pandas vs pyarrow write?

pandas to_parquet uses pyarrow/fastparquet under hood.
pq.write_table when you already have Arrow Table.

Polars and pandas interchange?

Arrow bridge: pl.from_pandas / to_pandas zero-copy when dtypes align.

Performance & Memory - column pruning
PySpark - distributed Parquet writes
Data Engineering Basics - partition overwrite
Data Validation & Quality - schema gates

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

File Formats

Search Documentation

File Formats