Data & Model Versioning (DVC)
DVC versions large datasets and model artifacts alongside git code. Pipeline stages create reproducible DAGs that rerun only when inputs change.
Recipe
dvc init
dvc add data/train.csv
git add data/train.csv.dvc .gitignore
dvc push # upload to remote storage
dvc pull # download on another machineWorking Example
# dvc.yaml
stages:
prepare:
cmd: python src/prepare.py data/raw data/prepared
deps:
- src/prepare.py
- data/raw
outs:
- data/prepared/train.csv
train:
cmd: python src/train.py data/prepared/train.csv models/model.joblib
deps:
- src/train.py
- data/prepared/train.csv
outs:
- models/model.joblib
metrics:
- metrics.jsondvc repro train # run pipeline, skip unchanged stages
dvc metrics show # compare metrics across git branchesGotchas
- Committing large files to git - repo bloat. Fix:
dvc add+ remote storage. - Not configuring remote - data stays local only. Fix:
dvc remote add -d storage s3://bucket/dvc. - Manual pipeline steps - not reproducible. Fix: define stages in
dvc.yaml.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| DVC | Git-centric ML teams | Non-git workflows |
| LakeFS | Data lake versioning | Simple CSV projects |
| W&B Artifacts | W&B ecosystem | Self-hosted data control |
| Manual S3 versioning | Minimal tooling | Need pipeline DAGs |
FAQs
DVC vs git-lfs?
DVC integrates with cloud storage and ML pipelines; git-lfs stores in git hosting.How does dvc repro work?
Checks stage deps/outs hashes; reruns only changed stages and downstream.Remote storage?
S3, GCS, Azure, SSH - `dvc push`/`dvc pull` syncs.Metrics tracking?
`metrics.json` tracked by DVC; compare across commits.Params?
`params.yaml` tracked; changes trigger downstream stages.With MLflow?
DVC versions data; MLflow tracks experiments - complementary.Cache?
DVC caches outputs locally; avoids redundant recomputation.Branching data?
Different git branches can point to different .dvc file versions.Lock file?
dvc.lock pins exact stage output hashes.Monorepo?
DVC works per-project subdirectory.Secrets?
Remote credentials via env vars or config.local (gitignored).CI/CD?
`dvc pull` in CI to fetch data; `dvc repro` to train.Related
- Experiment Tracking
- Model Registries & Versioning
- MLOps Best Practices
- Git
- Machine Learning Best Practices
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.