Drop or hash identifiers before notebooks leave secure environments. Log column lists, not row samples, in job logs.

Data Analysis Best Practices

Q: How do I version datasets?

Content-addressed Parquet paths (dt=2025-01-15/) plus manifest JSON. Never overwrite inputs - write new partitions.

Reproducible, vectorized, well-typed analysis habits that prevent silent wrong numbers in notebooks and production jobs.

How to Use This List

Walk A through D before shipping a report or scheduled job.
Check items in PR review for any change touching ingest, joins, or aggregates.
Revisit after incidents (wrong KPI, OOM, duplicate rows) and add team-specific rules.

A - Ingest & Types

Read with explicit dtypes. Pass dtype, parse_dates, and usecols to read_csv - inference hides object columns.
Count rows and nulls immediately. len(df), df.isna().sum(), and df.duplicated(subset=keys).sum() before transforms.
Store timestamps in UTC. pd.to_datetime(..., utc=True) at load; convert for display only.
Prefer Arrow strings and categoricals. string[pyarrow] and category cut RAM on text keys in pandas 2.2+.
Document NA imputation. Means/medians are modeling choices - record them in notebook or job metadata.

B - Transforms & Vectorization

Use .loc for assignments. Never chain df[mask]["col"] = x - assign in one .loc call.
Prefer built-in agg over apply. groupby.agg and Polars expressions beat Python per-row UDFs.
Set observed=True on categoricals. Skip unused category levels in pandas 2.x groupby/pivot.
Normalize string keys before joins. str.strip() and consistent casing prevent orphan merges.
Clip or flag outliers; do not silently drop. Domain caps need stakeholder sign-off.

C - Joins & Aggregates

Assert row counts around merges. Many-to-many explosions are the top KPI bug.
Use validate= on merges. many_to_one and one_to_one catch cardinality mistakes early.
Add indicator=True on left joins. Quantify unmatched dimension rows before filling NA with zero.
Sort time index before resample/rolling. Unsorted timestamps produce garbage windows.
Name aggregations explicitly. Named agg(total=("revenue", "sum")) beats mystery MultiIndex columns.

D - Performance & Reproducibility

Measure memory with deep=True. Shallow memory_usage lies on object columns.
Chunk or scan files that exceed RAM. chunksize, Parquet column pruning, or Polars scan_parquet.
Write intermediate Parquet with schema. Faster reruns than re-parsing CSV.
Pin deps with uv lockfiles. pandas 2.2+ and Polars 1.x behavior should not drift between laptop and CI.
Seed random sampling. sample(random_state=42) and numpy.random.default_rng for auditability.

FAQs

What is the single highest-impact habit?

Row-count and key-uniqueness checks at ingest and after every merge.
Most production KPI bugs are cardinality, not math.

When is pandas apply acceptable?

Prototypes on tiny samples only.
Replace with vectorized ops before scheduling the job.

Should notebooks be the source of truth?

Notebooks explore; scheduled scripts/modules with tests ship KPIs.
Extract functions once logic stabilizes.

How do I test an analysis?

Fixture CSV/Parquet snippets in pytest with expected aggregates.
Assert dtypes dict and row counts, not only final numbers.

When migrate to Polars?

After pandas dtype tuning and chunking still miss SLA or RAM budget.
Benchmark one hot path before wholesale rewrite.

How do I share results with stakeholders?

Export charts and tables with filters documented.
Never hand-edit Excel after export - change the script.

What about SQL instead of pandas?

DuckDB/SQL excels when data already lives in warehouse tables.
pandas still wins for small local files and rich Python ecosystem.

How do I handle PII?

Drop or hash identifiers before notebooks leave secure environments.
Log column lists, not row samples, in job logs.

Should I use float32?

For ML features and large numeric matrices when precision trade-off is documented.
Use float64 for financial totals unless measured safe.

How do I version datasets?

Content-addressed Parquet paths (dt=2025-01-15/) plus manifest JSON.
Never overwrite inputs - write new partitions.

Data Analysis Basics - intro tour
Cleaning & Transforming Data - dtype and NA policy
Joins & Merges - cardinality checks
Performance & Memory - chunking and Arrow
Data Validation & Quality - schema gates

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Data Analysis Best Practices

Reproducible, vectorized, well-typed analysis habits that prevent silent wrong numbers in notebooks and production jobs.

How to Use This List

Walk A through D before shipping a report or scheduled job.
Check items in PR review for any change touching ingest, joins, or aggregates.
Revisit after incidents (wrong KPI, OOM, duplicate rows) and add team-specific rules.

A - Ingest & Types

Read with explicit dtypes. Pass dtype, parse_dates, and usecols to read_csv - inference hides object columns.
Count rows and nulls immediately. len(df), df.isna().sum(), and df.duplicated(subset=keys).sum() before transforms.
Store timestamps in UTC. pd.to_datetime(..., utc=True) at load; convert for display only.
Prefer Arrow strings and categoricals. string[pyarrow] and category cut RAM on text keys in pandas 2.2+.
Document NA imputation. Means/medians are modeling choices - record them in notebook or job metadata.

B - Transforms & Vectorization

Use .loc for assignments. Never chain df[mask]["col"] = x - assign in one .loc call.
Prefer built-in agg over apply. groupby.agg and Polars expressions beat Python per-row UDFs.
Set observed=True on categoricals. Skip unused category levels in pandas 2.x groupby/pivot.
Normalize string keys before joins. str.strip() and consistent casing prevent orphan merges.
Clip or flag outliers; do not silently drop. Domain caps need stakeholder sign-off.

C - Joins & Aggregates

Assert row counts around merges. Many-to-many explosions are the top KPI bug.
Use validate= on merges. many_to_one and one_to_one catch cardinality mistakes early.
Add indicator=True on left joins. Quantify unmatched dimension rows before filling NA with zero.
Sort time index before resample/rolling. Unsorted timestamps produce garbage windows.
Name aggregations explicitly. Named agg(total=("revenue", "sum")) beats mystery MultiIndex columns.

D - Performance & Reproducibility

Measure memory with deep=True. Shallow memory_usage lies on object columns.
Chunk or scan files that exceed RAM. chunksize, Parquet column pruning, or Polars scan_parquet.
Write intermediate Parquet with schema. Faster reruns than re-parsing CSV.
Pin deps with uv lockfiles. pandas 2.2+ and Polars 1.x behavior should not drift between laptop and CI.
Seed random sampling. sample(random_state=42) and numpy.random.default_rng for auditability.

FAQs

What is the single highest-impact habit?

Row-count and key-uniqueness checks at ingest and after every merge.
Most production KPI bugs are cardinality, not math.

When is pandas apply acceptable?

Prototypes on tiny samples only.
Replace with vectorized ops before scheduling the job.

Should notebooks be the source of truth?

Notebooks explore; scheduled scripts/modules with tests ship KPIs.
Extract functions once logic stabilizes.

How do I test an analysis?

Fixture CSV/Parquet snippets in pytest with expected aggregates.
Assert dtypes dict and row counts, not only final numbers.

When migrate to Polars?

After pandas dtype tuning and chunking still miss SLA or RAM budget.
Benchmark one hot path before wholesale rewrite.

How do I share results with stakeholders?

Export charts and tables with filters documented.
Never hand-edit Excel after export - change the script.

What about SQL instead of pandas?

DuckDB/SQL excels when data already lives in warehouse tables.
pandas still wins for small local files and rich Python ecosystem.

How do I handle PII?

Drop or hash identifiers before notebooks leave secure environments.
Log column lists, not row samples, in job logs.

Should I use float32?

For ML features and large numeric matrices when precision trade-off is documented.
Use float64 for financial totals unless measured safe.

How do I version datasets?

Content-addressed Parquet paths (dt=2025-01-15/) plus manifest JSON.
Never overwrite inputs - write new partitions.

Data Analysis Basics - intro tour
Cleaning & Transforming Data - dtype and NA policy
Joins & Merges - cardinality checks
Performance & Memory - chunking and Arrow
Data Validation & Quality - schema gates

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Data Analysis Best Practices

Search Documentation

Data Analysis Best Practices