Machine Learning Best Practices

A checklist of rules for building reproducible, leak-free classical ML models. Walk it before every training run and during code review.

How to Use This List

• Apply rules A-C before writing training code.
• Rules D-F govern evaluation and model selection.
• Rules G-H cover deployment and maintenance.
• Revisit after every data schema change or new feature source.

A - Data Hygiene

• Split data before preprocessing. Fit scalers, encoders, and feature selectors on training folds only - never on the full dataset or test set.
• Lock a held-out test set. Touch it once for final evaluation after all model and hyperparameter decisions are made.
• Use stratified splits for classification. Preserve class ratios when any class is below 10% of rows.
• Split by group for repeated entities. Use GroupKFold when multiple rows share a user, patient, or session ID.
• Document data provenance. Record source, extraction date, filters applied, and row counts in an experiment log.

B - Feature Engineering

• Exclude post-outcome columns. Drop features only known after the prediction target (leakage).
• Wrap transforms in a Pipeline. Bundle preprocessing with the model so CV and deployment apply identical steps.
• Handle unknown categories at inference. Use OneHotEncoder(handle_unknown="ignore") or native categorical support in boosting libraries.
• Impute before scaling. Missing values must be filled before StandardScaler or distance-based models.
• Enforce column order at inference. Pass X[expected_columns] before predict to prevent silent misalignment.

C - Modeling

• Start with a simple baseline. Logistic regression or random forest before gradient boosting or deep learning.
• Set random seeds everywhere. random_state=42 on splits, models, and search for reproducible experiments.
• Match algorithms to problem type. Regression metrics for continuous targets; classification metrics for categories.
• Use class weights or SMOTE for imbalance. Do not trust accuracy when the positive class is rare.
• Serialize the full pipeline. joblib.dump(pipe, ...) - not just the classifier step.

D - Evaluation

• Pick metrics aligned with business cost. PR-AUC for rare positives; MAE for interpretable regression errors.
• Tune thresholds on validation data. Default 0.5 is rarely optimal for imbalanced classification.
• Report per-class metrics. A single F1 or accuracy hides failures on minority classes.
• Compare against the baseline. Every new model must beat the simple benchmark to justify complexity.
• Check train vs test gap. Large gap signals overfitting - reduce capacity or add regularization.

E - Hyperparameter Tuning

• Search inside cross-validation. Never pick hyperparameters using test set performance.
• Match scoring to deployment metric. Optimize f1, roc_auc, or neg_mean_absolute_error - not default accuracy.
• Cap search budget. Use Optuna pruning or random search instead of exhaustive grids on 5+ parameters.
• Log every trial. Record params, CV score, duration, and data version for auditability.

F - Reproducibility

• Pin package versions. Record scikit-learn, pandas, and boosting library versions in the artifact metadata.
• Version training data. Hash the training set or use DVC to tie models to exact data snapshots.
• Track experiments. Log params, metrics, and artifacts in MLflow, W&B, or an equivalent system.
• Write tests for inference. Assert pipe.predict(sample) returns expected shape and dtype.

FAQs

project/
  data/           # raw and processed (gitignored or DVC)
  src/train.py    # training script
  src/predict.py  # inference entrypoint
  models/         # serialized pipelines
  tests/          # inference smoke tests

def test_pipeline_predicts(sample_row):
    pred = pipe.predict(sample_row)
    assert pred.shape == (1,)

Related

• Pipelines - leak-free preprocessing
• Datasets & Splits - proper splitting strategies
• Model Evaluation - metric selection
• Experiment Tracking - logging runs
• Monitoring and Drift - production health

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.