MLOps Best Practices

Q: Biggest MLOps mistake?

Deploying notebook code without tests, versioning, or monitoring.

Q: Notebooks in production?

No - refactor to scripts with CLI args and tests.

Q: Minimum monitoring?

Health check, error rate, latency, prediction logging.

Q: When to use MLflow?

Any project with more than one experiment or team member.

Q: FastAPI enough for serving?

Yes for most teams until throughput demands Triton.

Q: How often to retrain?

On drift alert or scheduled (weekly/monthly) based on domain velocity.

Q: CI/CD for ML?

CI: tests + eval metrics. CD: deploy Staging model; promote to Production.

Q: Feature store needed?

When batch and streaming paths must share exact feature computation.

Q: Model cards?

Document intended use, limitations, training data, and bias risks.

Q: Security?

Auth on inference API; no PII in logs; scan container images.

Rules for taking models from experiment to reliable production service.

How to Use This List

Rules A-B cover reproducibility and tracking.
Rules C-D govern serving and inference.
Rules E-F cover monitoring and operations.

A - Reproducibility

Version training data with DVC or hash. Reproduce any model from code + data version.
Pin library versions in training environment. Record in MLflow run or container image.
Use training scripts, not notebooks. Notebooks for exploration; scripts for production pipelines.
Set random seeds and log them. Reproducible experiments across runs.
Save the full pipeline artifact. Preprocessing + model in one serializable object.

B - Experiment Tracking

Log every training run. Params, metrics, artifacts, data version, git commit.
Name runs descriptively. Include model type, data version, and key hyperparameter.
Compare runs before selecting a model. Do not deploy the last run by default.
Register best models in a registry. Staging -> Production promotion workflow.
Link registry version to training run. Full lineage from prediction back to data.

C - Serving

Load model once at startup. Lifespan handler or module-level load; never per request.
Validate inputs with Pydantic. Type, range, and required field checks.
Return model version with predictions. Traceability for debugging.
Expose /health endpoint. Model loaded, dependencies reachable.
Write inference smoke tests. Known input -> expected output shape and approximate value.

D - Inference Quality

Match training preprocessing in serving. Same pipeline artifact or shared code module.
Enforce feature column order. Named fields, not positional lists.
Log prediction metadata. Timestamp, model version, input hash (not raw PII).
Define latency budgets. p99 target; profile before choosing serving framework.
Plan rollback. Keep previous model version deployable in <5 minutes.

E - Monitoring

Save reference data distribution at deploy. Baseline for drift detection.
Monitor feature drift weekly. KS test, PSI, or Evidently reports.
Track prediction distribution. Sudden shifts indicate upstream data issues.
Alert on error rate and latency. Same observability as any production service.
Retrain on drift or metric degradation. Automated trigger with human approval gate.

F - Cost & Infrastructure

Right-size compute. CPU for sklearn; GPU only when training/inference requires it.
Use spot instances for training. Checkpoint to survive preemption.
Calculate API vs self-hosted break-even. Before investing in GPU infrastructure.
Auto-scale inference replicas. Scale to zero for dev; min replicas for production.
Tag cloud resources by project. Cost attribution and cleanup.

FAQs

Biggest MLOps mistake?

Deploying notebook code without tests, versioning, or monitoring.

Notebooks in production?

No - refactor to scripts with CLI args and tests.

Minimum monitoring?

Health check, error rate, latency, prediction logging.

When to use MLflow?

Any project with more than one experiment or team member.

FastAPI enough for serving?

Yes for most teams until throughput demands Triton.

How often to retrain?

On drift alert or scheduled (weekly/monthly) based on domain velocity.

CI/CD for ML?

CI: tests + eval metrics. CD: deploy Staging model; promote to Production.

Feature store needed?

When batch and streaming paths must share exact feature computation.

Model cards?

Document intended use, limitations, training data, and bias risks.

Security?

Auth on inference API; no PII in logs; scan container images.

LLM MLOps differences?

Add prompt versioning, eval sets, and token cost tracking to standard MLOps.

Team handoff?

README with train/serve commands, model registry location, and on-call runbook.

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

MLOps Best Practices

Rules for taking models from experiment to reliable production service.

How to Use This List

Rules A-B cover reproducibility and tracking.
Rules C-D govern serving and inference.
Rules E-F cover monitoring and operations.

A - Reproducibility

Version training data with DVC or hash. Reproduce any model from code + data version.
Pin library versions in training environment. Record in MLflow run or container image.
Use training scripts, not notebooks. Notebooks for exploration; scripts for production pipelines.
Set random seeds and log them. Reproducible experiments across runs.
Save the full pipeline artifact. Preprocessing + model in one serializable object.

B - Experiment Tracking

Log every training run. Params, metrics, artifacts, data version, git commit.
Name runs descriptively. Include model type, data version, and key hyperparameter.
Compare runs before selecting a model. Do not deploy the last run by default.
Register best models in a registry. Staging -> Production promotion workflow.
Link registry version to training run. Full lineage from prediction back to data.

C - Serving

Load model once at startup. Lifespan handler or module-level load; never per request.
Validate inputs with Pydantic. Type, range, and required field checks.
Return model version with predictions. Traceability for debugging.
Expose /health endpoint. Model loaded, dependencies reachable.
Write inference smoke tests. Known input -> expected output shape and approximate value.

D - Inference Quality

Match training preprocessing in serving. Same pipeline artifact or shared code module.
Enforce feature column order. Named fields, not positional lists.
Log prediction metadata. Timestamp, model version, input hash (not raw PII).
Define latency budgets. p99 target; profile before choosing serving framework.
Plan rollback. Keep previous model version deployable in <5 minutes.

E - Monitoring

Save reference data distribution at deploy. Baseline for drift detection.
Monitor feature drift weekly. KS test, PSI, or Evidently reports.
Track prediction distribution. Sudden shifts indicate upstream data issues.
Alert on error rate and latency. Same observability as any production service.
Retrain on drift or metric degradation. Automated trigger with human approval gate.

F - Cost & Infrastructure

Right-size compute. CPU for sklearn; GPU only when training/inference requires it.
Use spot instances for training. Checkpoint to survive preemption.
Calculate API vs self-hosted break-even. Before investing in GPU infrastructure.
Auto-scale inference replicas. Scale to zero for dev; min replicas for production.
Tag cloud resources by project. Cost attribution and cleanup.

FAQs

Biggest MLOps mistake?

Deploying notebook code without tests, versioning, or monitoring.

Notebooks in production?

No - refactor to scripts with CLI args and tests.

Minimum monitoring?

Health check, error rate, latency, prediction logging.

When to use MLflow?

Any project with more than one experiment or team member.

FastAPI enough for serving?

Yes for most teams until throughput demands Triton.

How often to retrain?

On drift alert or scheduled (weekly/monthly) based on domain velocity.

CI/CD for ML?

CI: tests + eval metrics. CD: deploy Staging model; promote to Production.

Feature store needed?

When batch and streaming paths must share exact feature computation.

Model cards?

Document intended use, limitations, training data, and bias risks.

Security?

Auth on inference API; no PII in logs; scan container images.

LLM MLOps differences?

Add prompt versioning, eval sets, and token cost tracking to standard MLOps.

Team handoff?

README with train/serve commands, model registry location, and on-call runbook.

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

MLOps Best Practices

Search Documentation

MLOps Best Practices