Git for Data/Notebooks

Q: nbstripout or clear manually?

Automate with pre-commit - manual clear is forgotten on every commit.

Q: LFS or DVC?

LFS for few large binaries in repo; DVC for versioned dataset pipelines and repro experiments.

Q: Review notebook PRs?

Use GitHub notebook diff or review paired .py from Jupytext for readable changes.

Q: CI execute notebooks?

papermill or nbclient with pinned data fixtures - not live prod data.

Q: ipynb in library package?

Avoid - move logic to src/ modules; keep notebooks in notebooks/ or docs/.

Q: pre-commit slow on huge repo?

Scope nbstripout to notebooks/ path only in hook files regex.

Jupyter notebooks, parquet exports, and model checkpoints need Git policies different from application code - nbstripout, Git LFS, and data versioning tools keep PRs reviewable and repos small.

Recipe

pip install nbstripout
nbstripout --install --attributes
git add .gitattributes notebooks/analysis.ipynb
git commit -m "chore: strip notebook outputs on commit"

When to reach for this:

Data science repos with .ipynb exploration
ML teams sharing notebooks that must diff in PRs
Parquet/model files too large for normal git objects
Compliance requiring PII never in git history

Working Example

# .pre-commit-config.yaml excerpt
repos:
  - repo: https://github.com/kynan/nbstripout
    rev: 0.7.1
    hooks:
      - id: nbstripout
 
# Track large artifacts with LFS
git lfs install
git lfs track "*.parquet"
git lfs track "models/*.pt"
git add .gitattributes
git commit -m "chore: LFS track parquet and model weights"

# Prefer referencing data by URI in notebooks
DATA_URI = "s3://acme-analytics/processed/orders/dt=2026-07-08/"
# not: pd.read_csv("data/orders_10gb.csv")  # committed to git

What this demonstrates:

nbstripout hook clears outputs before commit
LFS pointers replace bloated blobs in git history
Notebooks reference external data paths
.gitattributes documents filter rules

Deep Dive

How It Works

Notebook JSON - Stores code, metadata, and outputs - outputs change every run causing huge diffs.
nbstripout - Git clean/smudge filter removing outputs and execution_count.
Git LFS - Stores large files on LFS server; repo holds pointer text files.
DVC - Version data in S3/GCS with small .dvc files in git.

Strategy Picker

Asset	Strategy
Notebooks	nbstripout + optional Jupytext `.py`
10MB-5GB files	Git LFS with quota monitoring
Datasets	DVC, cloud partition paths, catalog
Secrets in notebooks	Never - use env + pydantic-settings

Python Notes

# Jupytext paired notebook
jupytext --set-formats ipynb,py:percent analysis.ipynb
git add analysis.ipynb analysis.py

Gotchas

Committing outputs once - History forever bloated even after nbstripout. Fix: filter-repo or BFG if leaked; prevent with hooks.
LFS bandwidth costs - CI clones download LFS. Fix: GIT_LFS_SKIP_SMUDGE=1 in CI that does not need data.
PII in committed CSV - GDPR incident. Fix: synthetic fixtures in git; prod data in vault bucket.
Binary merge conflicts on notebooks - JSON hell. Fix: strip outputs; one editor per notebook file.
DVC lock out of sync - Wrong data version pulled. Fix: commit .dvc with code; dvc pull in CI docs.
Pickle in notebooks - Arbitrary code execution risk. Fix: parquet/JSON for interchange.

Alternatives

Alternative	Use When	Don't Use When
Papermill parameters	Scheduled notebook runs	Library code belongs in `.py` modules
Marimo / percent scripts	Git-friendly source first	Team standardized on classic Jupyter
SageMaker / Colab only	No local git for notebooks	Org requires PR review on git
Quarto	Report publishing pipeline	Quick internal EDA only

FAQs

nbstripout or clear manually?

Automate with pre-commit - manual clear is forgotten on every commit.

LFS or DVC?

LFS for few large binaries in repo; DVC for versioned dataset pipelines and repro experiments.

Review notebook PRs?

Use GitHub notebook diff or review paired .py from Jupytext for readable changes.

CI execute notebooks?

papermill or nbclient with pinned data fixtures - not live prod data.

ipynb in library package?

Avoid - move logic to src/ modules; keep notebooks in notebooks/ or docs/.

Git LFS free tier limits?

Monitor storage and bandwidth on GitHub - migrate big data to object store when exceeded.

Merge conflicts in ipynb?

Re-run nbstripout on both sides or choose one author's notebook and re-apply cells manually.

nbconvert outputs in git?

Do not commit HTML/PDF exports - generate in CI artifacts.

wandb artifacts vs git?

Store run artifacts in W&B/S3; git tracks only training commit SHA and config YAML.

pre-commit slow on huge repo?

Scope nbstripout to notebooks/ path only in hook files regex.

Data Engineering Basics - pipeline norms
Git Configuration - hooks setup
Merge Conflict Resolution Checklist - notebook conflicts
30 Data/ML Code Rules - team rules

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Recipe

pip install nbstripout
nbstripout --install --attributes
git add .gitattributes notebooks/analysis.ipynb
git commit -m "chore: strip notebook outputs on commit"

When to reach for this:

Data science repos with .ipynb exploration
ML teams sharing notebooks that must diff in PRs
Parquet/model files too large for normal git objects
Compliance requiring PII never in git history

Working Example

# .pre-commit-config.yaml excerpt
repos:
  - repo: https://github.com/kynan/nbstripout
    rev: 0.7.1
    hooks:
      - id: nbstripout
 
# Track large artifacts with LFS
git lfs install
git lfs track "*.parquet"
git lfs track "models/*.pt"
git add .gitattributes
git commit -m "chore: LFS track parquet and model weights"

# Prefer referencing data by URI in notebooks
DATA_URI = "s3://acme-analytics/processed/orders/dt=2026-07-08/"
# not: pd.read_csv("data/orders_10gb.csv")  # committed to git

What this demonstrates:

nbstripout hook clears outputs before commit
LFS pointers replace bloated blobs in git history
Notebooks reference external data paths
.gitattributes documents filter rules

Deep Dive

How It Works

Notebook JSON - Stores code, metadata, and outputs - outputs change every run causing huge diffs.
nbstripout - Git clean/smudge filter removing outputs and execution_count.
Git LFS - Stores large files on LFS server; repo holds pointer text files.
DVC - Version data in S3/GCS with small .dvc files in git.

Strategy Picker

Asset	Strategy
Notebooks	nbstripout + optional Jupytext `.py`
10MB-5GB files	Git LFS with quota monitoring
Datasets	DVC, cloud partition paths, catalog
Secrets in notebooks	Never - use env + pydantic-settings

Python Notes

# Jupytext paired notebook
jupytext --set-formats ipynb,py:percent analysis.ipynb
git add analysis.ipynb analysis.py

Gotchas

Committing outputs once - History forever bloated even after nbstripout. Fix: filter-repo or BFG if leaked; prevent with hooks.
LFS bandwidth costs - CI clones download LFS. Fix: GIT_LFS_SKIP_SMUDGE=1 in CI that does not need data.
PII in committed CSV - GDPR incident. Fix: synthetic fixtures in git; prod data in vault bucket.
Binary merge conflicts on notebooks - JSON hell. Fix: strip outputs; one editor per notebook file.
DVC lock out of sync - Wrong data version pulled. Fix: commit .dvc with code; dvc pull in CI docs.
Pickle in notebooks - Arbitrary code execution risk. Fix: parquet/JSON for interchange.

Alternatives

Alternative	Use When	Don't Use When
Papermill parameters	Scheduled notebook runs	Library code belongs in `.py` modules
Marimo / percent scripts	Git-friendly source first	Team standardized on classic Jupyter
SageMaker / Colab only	No local git for notebooks	Org requires PR review on git
Quarto	Report publishing pipeline	Quick internal EDA only

FAQs

nbstripout or clear manually?

Automate with pre-commit - manual clear is forgotten on every commit.

LFS or DVC?

LFS for few large binaries in repo; DVC for versioned dataset pipelines and repro experiments.

Review notebook PRs?

Use GitHub notebook diff or review paired .py from Jupytext for readable changes.

CI execute notebooks?

papermill or nbclient with pinned data fixtures - not live prod data.

ipynb in library package?

Avoid - move logic to src/ modules; keep notebooks in notebooks/ or docs/.

Git LFS free tier limits?

Monitor storage and bandwidth on GitHub - migrate big data to object store when exceeded.

Merge conflicts in ipynb?

Re-run nbstripout on both sides or choose one author's notebook and re-apply cells manually.

nbconvert outputs in git?

Do not commit HTML/PDF exports - generate in CI artifacts.

wandb artifacts vs git?

Store run artifacts in W&B/S3; git tracks only training commit SHA and config YAML.

pre-commit slow on huge repo?

Scope nbstripout to notebooks/ path only in hook files regex.

Data Engineering Basics - pipeline norms
Git Configuration - hooks setup
Merge Conflict Resolution Checklist - notebook conflicts
30 Data/ML Code Rules - team rules

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Git for Data/Notebooks

Search Documentation

Git for Data/Notebooks