Classical ML Highlights Summary

Every highlight bullet from the 12 pages in this section, gathered on one page and grouped by the page it came from.

The Classical ML Mental Model

• fit learns parameters from data; predict/transform apply them - mixing the two is how leakage happens
• A held-out test set only measures generalization if it was never touched during fitting
• Validation exists because the test set can only be used once without becoming training-by-proxy
• Bias and variance trade off against each other - fixing one usually costs the other
• Leakage inflates validation metrics while quietly guaranteeing worse production performance

Machine Learning Basics

• Split data before any preprocessing to avoid leakage
• Pipelines bundle transforms and models for reproducible training
• Pick metrics that match the business problem, not model convenience
• Set random seeds for reproducible experiments

Datasets & Splits

• Split before preprocessing to prevent data leakage
• Use stratified splits for imbalanced classification
• Cross-validation estimates performance without touching the test set
• Group splits prevent the same entity appearing in train and test

Feature Engineering

• ColumnTransformer applies different transforms per column type
• Fit preprocessors on training data only
• OneHotEncoder handles unknown categories at inference
• Polynomial features increase capacity but risk overfitting

Pipelines

• Pipelines prevent preprocessing leakage across CV folds
• Serialize the full pipeline for deployment, not just the model
• Step names enable hyperparameter search with double underscores
• FeatureUnion combines parallel transform branches

Supervised Learning

• Start with a simple baseline before complex models
• Linear models for interpretability, trees for interactions
• Match the algorithm to problem type: regression vs classification
• Regularization controls overfitting in linear models

Model Evaluation

• Pick metrics aligned with business cost, not convenience
• ROC-AUC measures ranking; PR-AUC is better for rare positives
• Confusion matrices reveal which classes confuse the model
• Calibration matters when probabilities drive decisions

Hyperparameter Tuning

• Search inside cross-validation, never on the test set
• Random search beats grid search in high dimensions
• Optuna prunes unpromising trials early
• Log every trial for reproducibility

Unsupervised Learning

• Scale features before distance-based clustering
• Choose cluster count with elbow method or silhouette score
• PCA reduces dimensions while preserving variance
• Unsupervised labels need domain validation

Imbalanced Data

• Accuracy misleads when one class dominates
• SMOTE inside CV pipelines prevents synthetic leakage
• Class weights are simpler than resampling for many cases
• Tune the decision threshold for business-aligned recall

Gradient Boosting

• Gradient boosting wins on most tabular competitions
• Early stopping prevents overfitting on extra trees
• Native categorical support in LightGBM and XGBoost 2.0+
• scale_pos_weight handles class imbalance

Machine Learning Best Practices

• Split before any preprocessing or feature selection
• Pipeline everything that learns from data
• Pick metrics that match business cost
• Version data, code, and model artifacts together