Machine Learning Best Practices

A checklist of rules for building reproducible, leak-free classical ML models. Walk it before every training run and during code review.

How to Use This List

Apply rules A-C before writing training code.
Rules D-F govern evaluation and model selection.
Rules G-H cover deployment and maintenance.
Revisit after every data schema change or new feature source.

A - Data Hygiene

Split data before preprocessing. Fit scalers, encoders, and feature selectors on training folds only - never on the full dataset or test set.
Lock a held-out test set. Touch it once for final evaluation after all model and hyperparameter decisions are made.
Use stratified splits for classification. Preserve class ratios when any class is below 10% of rows.
Split by group for repeated entities. Use GroupKFold when multiple rows share a user, patient, or session ID.
Document data provenance. Record source, extraction date, filters applied, and row counts in an experiment log.

B - Feature Engineering

Exclude post-outcome columns. Drop features only known after the prediction target (leakage).
Wrap transforms in a Pipeline. Bundle preprocessing with the model so CV and deployment apply identical steps.
Handle unknown categories at inference. Use OneHotEncoder(handle_unknown="ignore") or native categorical support in boosting libraries.
Impute before scaling. Missing values must be filled before StandardScaler or distance-based models.
Enforce column order at inference. Pass X[expected_columns] before predict to prevent silent misalignment.

C - Modeling

Start with a simple baseline. Logistic regression or random forest before gradient boosting or deep learning.
Set random seeds everywhere. random_state=42 on splits, models, and search for reproducible experiments.
Match algorithms to problem type. Regression metrics for continuous targets; classification metrics for categories.
Use class weights or SMOTE for imbalance. Do not trust accuracy when the positive class is rare.
Serialize the full pipeline. joblib.dump(pipe, ...) - not just the classifier step.

D - Evaluation

Pick metrics aligned with business cost. PR-AUC for rare positives; MAE for interpretable regression errors.
Tune thresholds on validation data. Default 0.5 is rarely optimal for imbalanced classification.
Report per-class metrics. A single F1 or accuracy hides failures on minority classes.
Compare against the baseline. Every new model must beat the simple benchmark to justify complexity.
Check train vs test gap. Large gap signals overfitting - reduce capacity or add regularization.

E - Hyperparameter Tuning

Search inside cross-validation. Never pick hyperparameters using test set performance.
Match scoring to deployment metric. Optimize f1, roc_auc, or neg_mean_absolute_error - not default accuracy.
Cap search budget. Use Optuna pruning or random search instead of exhaustive grids on 5+ parameters.
Log every trial. Record params, CV score, duration, and data version for auditability.

F - Reproducibility

Pin package versions. Record scikit-learn, pandas, and boosting library versions in the artifact metadata.
Version training data. Hash the training set or use DVC to tie models to exact data snapshots.
Track experiments. Log params, metrics, and artifacts in MLflow, W&B, or an equivalent system.
Write tests for inference. Assert pipe.predict(sample) returns expected shape and dtype.

FAQs

What is the single most common ML mistake?

Training preprocessors on the full dataset before splitting.
Inflates CV and test scores; model fails in production on new data.
Fix with Pipeline and proper split order.

When can I touch the test set?

Once, after all modeling decisions are final.
For a single unbiased generalization estimate.
If you tune on it, get a fresh test set.

Do I always need cross-validation?

Yes for model comparison and hyperparameter search on datasets under 100k rows.
Single split is acceptable for very large data with stable metrics.

Should tree models use scaling?

No - random forest and boosting are scale-invariant.
Still encode categoricals and handle missing values.

How do I prevent notebook experiments from diverging?

Move training code to scripts or modules.
Use random_state, pinned deps, and experiment tracking.
Review notebooks for cells that fit on all data.

What belongs in a model card?

Training data description, metrics, limitations, and bias risks.
Feature list, preprocessing steps, and intended use cases.
Version of sklearn and data snapshot hash.

When is SMOTE safe?

Inside imblearn.pipeline.Pipeline after the train split.
On dense numeric features with moderate imbalance.
Not on the test set, ever.

How do I review a teammate's ML PR?

Check split order, pipeline usage, and metric choice.
Verify test set is not used in tuning.
Run their training script and compare metrics.

What is the minimum viable ML project structure?

project/
  data/           # raw and processed (gitignored or DVC)
  src/train.py    # training script
  src/predict.py  # inference entrypoint
  models/         # serialized pipelines
  tests/          # inference smoke tests

When should I move from sklearn to deep learning?

When data is images, audio, text, or long sequences.
Tabular data under 100k rows rarely benefits from neural nets.
Try gradient boosting first on structured data.

How often should I retrain?

When monitoring detects data drift or metric degradation.
On a schedule matching how fast the world changes.
Not on every new row - batch retraining is usually enough.

What tests should inference code have?

def test_pipeline_predicts(sample_row):
    pred = pipe.predict(sample_row)
    assert pred.shape == (1,)

Smoke test with a known input-output pair.
Assert column names match training schema.

Pipelines - leak-free preprocessing
Datasets & Splits - proper splitting strategies
Model Evaluation - metric selection
Experiment Tracking - logging runs
Monitoring and Drift - production health

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Machine Learning Best Practices

A checklist of rules for building reproducible, leak-free classical ML models. Walk it before every training run and during code review.

How to Use This List

Apply rules A-C before writing training code.
Rules D-F govern evaluation and model selection.
Rules G-H cover deployment and maintenance.
Revisit after every data schema change or new feature source.

A - Data Hygiene

Split data before preprocessing. Fit scalers, encoders, and feature selectors on training folds only - never on the full dataset or test set.
Lock a held-out test set. Touch it once for final evaluation after all model and hyperparameter decisions are made.
Use stratified splits for classification. Preserve class ratios when any class is below 10% of rows.
Split by group for repeated entities. Use GroupKFold when multiple rows share a user, patient, or session ID.
Document data provenance. Record source, extraction date, filters applied, and row counts in an experiment log.

B - Feature Engineering

Exclude post-outcome columns. Drop features only known after the prediction target (leakage).
Wrap transforms in a Pipeline. Bundle preprocessing with the model so CV and deployment apply identical steps.
Handle unknown categories at inference. Use OneHotEncoder(handle_unknown="ignore") or native categorical support in boosting libraries.
Impute before scaling. Missing values must be filled before StandardScaler or distance-based models.
Enforce column order at inference. Pass X[expected_columns] before predict to prevent silent misalignment.

C - Modeling

Start with a simple baseline. Logistic regression or random forest before gradient boosting or deep learning.
Set random seeds everywhere. random_state=42 on splits, models, and search for reproducible experiments.
Match algorithms to problem type. Regression metrics for continuous targets; classification metrics for categories.
Use class weights or SMOTE for imbalance. Do not trust accuracy when the positive class is rare.
Serialize the full pipeline. joblib.dump(pipe, ...) - not just the classifier step.

D - Evaluation

Pick metrics aligned with business cost. PR-AUC for rare positives; MAE for interpretable regression errors.
Tune thresholds on validation data. Default 0.5 is rarely optimal for imbalanced classification.
Report per-class metrics. A single F1 or accuracy hides failures on minority classes.
Compare against the baseline. Every new model must beat the simple benchmark to justify complexity.
Check train vs test gap. Large gap signals overfitting - reduce capacity or add regularization.

E - Hyperparameter Tuning

Search inside cross-validation. Never pick hyperparameters using test set performance.
Match scoring to deployment metric. Optimize f1, roc_auc, or neg_mean_absolute_error - not default accuracy.
Cap search budget. Use Optuna pruning or random search instead of exhaustive grids on 5+ parameters.
Log every trial. Record params, CV score, duration, and data version for auditability.

F - Reproducibility

Pin package versions. Record scikit-learn, pandas, and boosting library versions in the artifact metadata.
Version training data. Hash the training set or use DVC to tie models to exact data snapshots.
Track experiments. Log params, metrics, and artifacts in MLflow, W&B, or an equivalent system.
Write tests for inference. Assert pipe.predict(sample) returns expected shape and dtype.

FAQs

What is the single most common ML mistake?

Training preprocessors on the full dataset before splitting.
Inflates CV and test scores; model fails in production on new data.
Fix with Pipeline and proper split order.

When can I touch the test set?

Once, after all modeling decisions are final.
For a single unbiased generalization estimate.
If you tune on it, get a fresh test set.

Do I always need cross-validation?

Yes for model comparison and hyperparameter search on datasets under 100k rows.
Single split is acceptable for very large data with stable metrics.

Should tree models use scaling?

No - random forest and boosting are scale-invariant.
Still encode categoricals and handle missing values.

How do I prevent notebook experiments from diverging?

Move training code to scripts or modules.
Use random_state, pinned deps, and experiment tracking.
Review notebooks for cells that fit on all data.

What belongs in a model card?

Training data description, metrics, limitations, and bias risks.
Feature list, preprocessing steps, and intended use cases.
Version of sklearn and data snapshot hash.

When is SMOTE safe?

Inside imblearn.pipeline.Pipeline after the train split.
On dense numeric features with moderate imbalance.
Not on the test set, ever.

How do I review a teammate's ML PR?

Check split order, pipeline usage, and metric choice.
Verify test set is not used in tuning.
Run their training script and compare metrics.

What is the minimum viable ML project structure?

project/
  data/           # raw and processed (gitignored or DVC)
  src/train.py    # training script
  src/predict.py  # inference entrypoint
  models/         # serialized pipelines
  tests/          # inference smoke tests

When should I move from sklearn to deep learning?

When data is images, audio, text, or long sequences.
Tabular data under 100k rows rarely benefits from neural nets.
Try gradient boosting first on structured data.

How often should I retrain?

When monitoring detects data drift or metric degradation.
On a schedule matching how fast the world changes.
Not on every new row - batch retraining is usually enough.

What tests should inference code have?

def test_pipeline_predicts(sample_row):
    pred = pipe.predict(sample_row)
    assert pred.shape == (1,)

Smoke test with a known input-output pair.
Assert column names match training schema.

Pipelines - leak-free preprocessing
Datasets & Splits - proper splitting strategies
Model Evaluation - metric selection
Experiment Tracking - logging runs
Monitoring and Drift - production health

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Machine Learning Best Practices

Search Documentation

Machine Learning Best Practices