Search across all documentation pages
Every page in this section - experiment tracking, model registries, serving, monitoring - is a stage in one continuous lifecycle, and MLOps as a discipline exists because none of those stages work in isolation.
A model trained in a notebook is a liability, not an asset, until it has passed through a pipeline that makes it reproducible, comparable, deployable, and observable.
This page describes that lifecycle end to end, so that the more specific pages in this section read as chapters of one story rather than a list of unrelated tools.
A notebook is an excellent tool for exploration and a poor tool for production, because it conflates code, state, and output in a way that is nearly impossible to reproduce exactly later.
Experiment tracking is the first stage of the lifecycle, and it exists to solve exactly that problem: every training run logs its parameters, metrics, and resulting artifact to a system built for comparison, not just execution.
Without tracking, "which run produced our best model" becomes an unanswerable question after the fifth or sixth iteration, because notebook cell order and manual print statements do not survive a restarted kernel.
Once a run produces a model worth keeping, it needs to move into a model registry, which is the second stage.
A registry is not just file storage; it is a catalog of versions with lifecycle stages attached (commonly something like None, Staging, Production, Archived), so that "which model is live right now" is always a single, answerable query rather than a guess based on filenames.
The third stage is serving: exposing a registered model version behind an interface, typically an HTTP endpoint, so that other systems can request predictions without needing to know anything about how the model was trained.
The fourth and most frequently skipped stage is monitoring, which watches the live model's inputs and predictions over time to catch drift - the slow divergence between the data a model was trained on and the data it now sees in production.
A simple analogy: training a model is like calibrating a thermometer in a lab.
Experiment tracking is the lab notebook proving which calibration procedure was used.
The registry is the label on the thermometer stating which calibration version it carries.
Serving is putting the thermometer to work in the field.
Monitoring is periodically checking that the thermometer still reads true against a known reference, because the world it measures keeps changing even if the thermometer itself does not.
The lifecycle only works if each stage produces an artifact the next stage can consume unambiguously, which is why versioning threads through every stage rather than living in just one place.
An experiment run should version three things together: the code (git commit), the data (a hash or a DVC-tracked pointer), and the resulting parameters and metrics, because a model's behavior is a function of all three, and changing any one without recording it breaks reproducibility.
The registry's job is to take the winning run from experiment tracking and give it a stable identity - a name and version number - that serving can reference without caring which run produced it.
This decoupling matters mechanically: serving code should say "load models:/churn-predictor/Production," never "load the file my colleague trained last Tuesday," because the former survives retraining and redeployment while the latter does not.
Promotion between registry stages (Staging to Production) is where testing and validation actually get enforced - it is the mechanical gate between "a model that scored well offline" and "a model allowed to answer real requests."
Serving itself has its own internal trade-off: batch inference processes large volumes on a schedule with high throughput and no latency pressure, while online serving answers individual requests in real time with strict latency budgets, and a model's serving pattern should be chosen before, not after, the deployment architecture is built.
Monitoring closes the loop back to experiment tracking, because detecting drift is only useful if it triggers a new experiment, using fresh data, that produces a new candidate version to register and promote.
# The lifecycle as a chain of lookups, not a single script
run = experiment_tracker.best_run(metric="f1", experiment="churn")
version = registry.register(run.model_artifact, name="churn-predictor")
registry.promote(version, stage="Staging") # gate: offline validation
# ... validation passes ...
registry.promote(version, stage="Production") # gate: now serving reads this
serving_model = registry.load("churn-predictor", stage="Production")The point of this snippet is the shape, not the specific API: each stage hands off a small, stable reference (a run ID, a version number, a stage label) rather than passing raw files or notebook state directly between people.
At scale, the lifecycle has to handle two things a single-model demo never surfaces: multiple models competing for the same production slot, and models whose accuracy degrades without any code change at all.
Champion/challenger patterns extend the registry model beyond a single "Production" label, running a new candidate alongside the incumbent on a fraction of traffic before a full promotion, which turns promotion from a single risky cutover into a gradual, measured rollout.
Drift itself splits into at least two distinct failure modes that require different monitoring: data drift, where the distribution of incoming features shifts (new customer segment, seasonal behavior change), and concept drift, where the relationship between features and the true label changes even if feature distributions look stable (fraud patterns evolving, for example).
Neither kind of drift raises an exception or a 500 error - the model keeps returning confident predictions - which is exactly why monitoring has to be a deliberate, separate stage rather than something you assume error logs will catch.
GPU-backed training and serving introduce a cost dimension that CPU-only lifecycles do not have to reason about as carefully: idle GPU capacity between training runs, and serving infrastructure that must be right-sized against genuinely bursty traffic rather than provisioned for peak at all times.
Compliance-heavy environments add a further requirement on top of the base lifecycle: every promotion needs an audit trail linking a live production version back through its registry entry, its experiment run, and the exact data snapshot it trained on.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
| Manual notebook + file copy | Zero setup, fastest for a single analyst | No reproducibility, no rollback, no audit trail | One-off analysis, throwaway prototypes |
| Experiment tracking only (no registry) | Comparable runs, easy adoption | No clear "what is live" answer, manual promotion | Small teams, few models, low deployment frequency |
| Full lifecycle (tracking + registry + serving + monitoring) | Reproducible, auditable, safe rollback, drift-aware | Real infrastructure and process overhead | Any model retrained regularly or serving live traffic |
| Champion/challenger rollout | Safer promotions, real-traffic validation | Needs traffic-splitting infra and clear success metrics | High-stakes models, frequent retraining cadence |
Probably not the full stack. A single notebook and manual file saves are fine until more than one person touches the model, it gets retrained more than once, or it serves real traffic - at that point reproducibility problems appear quickly.
Each experiment run should record a git commit (code), a data hash or DVC pointer (data), and a registered artifact (model), so any past result can be reconstructed from three coordinates rather than tribal memory.
Batch inference runs on a schedule over a large dataset with relaxed latency and high throughput; online serving answers individual requests synchronously under a strict latency budget, which usually means a smaller, faster model or extra optimization work.
The model has no awareness that the world changed; it applies the same learned function to new inputs and returns a confident-looking prediction regardless of whether that function still matches reality, which is why drift needs explicit statistical monitoring, not error logs.
Data drift is a shift in the incoming feature distributions themselves; concept drift is a shift in the relationship between features and the true outcome, even if the features look statistically similar to before.
Use it when the cost of a bad promotion is high enough to justify validating a new version against a slice of real traffic before it fully replaces the incumbent, rather than relying only on offline metrics.
It's part of the same lifecycle - training and serving cadence directly determine GPU utilization, and idle or oversized GPU capacity is a recurring, measurable cost that the lifecycle's design choices directly influence.
Without a staging gate and validation step, a regression in data quality or a training bug ships straight to production with no checkpoint to catch it - the registry's stage transitions exist specifically to insert a review point.
The exact training data snapshot, the exact code version, the exact hyperparameters, and often the exact library versions - any one of these changing can change the resulting model even with identical code.
The stages are the same, but the cost and cadence differ - large models make retraining and champion/challenger rollouts more expensive, which pushes teams toward heavier monitoring to avoid unnecessary retrains.
Treating deployment as the finish line and skipping monitoring entirely, which means the first sign of drift is a business metric quietly declining weeks after the model actually stopped being accurate.
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.
Reviewed by Chris St. John·Last updated Jul 15, 2026