Supervised Learning

Q: What is the difference between regression and classification?

Regression outputs continuous values (price = 249.99). Classification outputs discrete labels (spam = yes/no). Multi-class classification has 3+ categories.

Q: When should I use logistic regression?

Strong linear baseline for classification. Interpretable coefficients for stakeholder review. Fast training on high-dimensional sparse text features.

Q: What does regularization alpha do in Ridge?

Higher alpha shrinks coefficients toward zero, reducing overfitting. alpha=0 is ordinary least squares. Tune with cross-validation (RidgeCV).

Q: How do tree models handle missing values?

sklearn RandomForest does not natively handle NaN - impute first. XGBoost and LightGBM learn optimal missing-value directions. Impute inside a pipeline for sklearn trees.

Q: What is the bias-variance tradeoff?

Simple models underfit (high bias); complex models overfit (high variance). Validation metrics reveal the sweet spot. Regularization and ensembling reduce variance.

Q: Can I use the same pipeline for train and production?

Yes - joblib.dump the fitted pipeline. Production calls pipe.predict(new_row) with the same column names.

Q: How do I handle multi-class classification?

Use average="macro" F1 when classes are imbalanced.

Q: What is a good first model for a new dataset?

Logistic regression or Ridge for baselines. RandomForest for a non-linear check without heavy tuning. Compare both before investing in gradient boosting.

Q: How do I interpret feature importance?

Tree models expose feature_importances_. Linear models use coefficient magnitudes (after scaling).

Q: Should I always use ensemble methods?

Ensembles win on tabular Kaggle-style data. Linear models win when interpretability and speed matter. Start simple, upgrade when validation justifies complexity.

Supervised learning maps input features to known labels. Regression predicts continuous values; classification assigns discrete categories. sklearn provides a consistent API across algorithms.

Recipe

Quick-reference recipe card - copy-paste ready.

from sklearn.linear_model import Ridge, LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingRegressor
 
# Classification
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
 
# Regression
reg = Ridge(alpha=1.0)
reg.fit(X_train, y_train)

When to reach for this:

Predicting a numeric outcome (price, demand, latency) - regression.
Assigning categories (churn, fraud, sentiment) - classification.
Tabular data with labeled training examples.
Baseline modeling before deep learning or gradient boosting tuning.

Working Example

"""supervised_learning.py - regression and classification with sklearn."""
from __future__ import annotations
 
import pandas as pd
from sklearn.datasets import fetch_california_housing, load_breast_cancer
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.metrics import mean_absolute_error, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
 
# --- Classification ---
cancer = load_breast_cancer(as_frame=True)
Xc, yc = cancer.data, cancer.target
Xc_train, Xc_test, yc_train, yc_test = train_test_split(Xc, yc, test_size=0.2, random_state=42)
 
clf_pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000, random_state=42)),
])
clf_pipe.fit(Xc_train, yc_train)
proba = clf_pipe.predict_proba(Xc_test)[:, 1]
print("classification AUC:", roc_auc_score(yc_test, proba))
 
# --- Regression ---
housing = fetch_california_housing(as_frame=True)
Xr, yr = housing.data, housing.target
Xr_train, Xr_test, yr_train, yr_test = train_test_split(Xr, yr, test_size=0.2, random_state=42)
 
reg = RandomForestRegressor(n_estimators=200, random_state=42, n_jobs=-1)
reg.fit(Xr_train, yr_train)
print("regression MAE:", mean_absolute_error(yr_test, reg.predict(Xr_test)))

What this demonstrates:

Same fit/predict API for regression and classification.
Scaling before logistic regression; tree models skip scaling.
Probability outputs for classification ranking metrics.
MAE as an interpretable regression metric (dollars, units).

Deep Dive

How It Works

Training minimizes a loss function over labeled examples (log loss, MSE, hinge loss).
Linear models learn weighted sums of features; fast and interpretable.
Tree ensembles partition feature space with axis-aligned splits; capture non-linear interactions.
Regularization (L1/L2) penalizes large weights to reduce overfitting.
Decision boundary complexity grows with model capacity - balance with validation metrics.

Algorithm Selection

Problem	Start With	Upgrade To
Binary classification	LogisticRegression	RandomForest, XGBoost
Multi-class	LogisticRegression (multinomial)	RandomForestClassifier
Regression (linear-ish)	Ridge	GradientBoostingRegressor
Regression (non-linear)	RandomForestRegressor	LightGBM
Small data, interpretability	Linear models	-
Large tabular, accuracy	Gradient boosting	-

Python Notes

from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
 
# SVM: strong with scaled features, slower on large data
svm = Pipeline([("scale", StandardScaler()), ("clf", SVC(probability=True))])
 
# k-NN: lazy learner, sensitive to scale and dimensionality
knn = Pipeline([("scale", StandardScaler()), ("clf", KNeighborsClassifier(n_neighbors=5))])

Gotchas

Using classification metrics for regression - accuracy on continuous targets is meaningless. Fix: MAE, RMSE, R-squared for regression.
Not scaling for SVM/k-NN/logistic regression - distance and gradient methods fail on unscaled features. Fix: StandardScaler in a pipeline.
Ignoring class imbalance - accuracy looks good while the model predicts the majority class. Fix: F1, ROC-AUC, or resampling.
Overfitting with deep trees - max_depth=None memorizes noise on small data. Fix: limit depth, increase min_samples_leaf, use CV.
Multicollinearity in linear regression - unstable coefficients with correlated features. Fix: Ridge regression or feature selection.
Predicting without predict_proba - some models need probability=True (SVC) for probability outputs. Fix: check estimator support before using proba.

Alternatives

Alternative	Use When	Don't Use When
Linear models	Interpretability, fast baseline	Complex non-linear interactions dominate
Random Forest	Robust default, little tuning	Need maximum accuracy on large tabular
Gradient boosting	Best tabular accuracy	Need fast training iteration or interpretability
Deep learning	Images, text, sequences	Small tabular with <10k rows

FAQs

What is the difference between regression and classification?

Regression outputs continuous values (price = 249.99).
Classification outputs discrete labels (spam = yes/no).
Multi-class classification has 3+ categories.

When should I use logistic regression?

Strong linear baseline for classification.
Interpretable coefficients for stakeholder review.
Fast training on high-dimensional sparse text features.

What does regularization alpha do in Ridge?

Higher alpha shrinks coefficients toward zero, reducing overfitting.
alpha=0 is ordinary least squares.
Tune with cross-validation (RidgeCV).

How do tree models handle missing values?

sklearn RandomForest does not natively handle NaN - impute first.
XGBoost and LightGBM learn optimal missing-value directions.
Impute inside a pipeline for sklearn trees.

What is the bias-variance tradeoff?

Simple models underfit (high bias); complex models overfit (high variance).
Validation metrics reveal the sweet spot.
Regularization and ensembling reduce variance.

Can I use the same pipeline for train and production?

Yes - joblib.dump the fitted pipeline.
Production calls pipe.predict(new_row) with the same column names.

How do I handle multi-class classification?

LogisticRegression(multi_class="multinomial", max_iter=1000)
# or
RandomForestClassifier()  # handles multi-class natively

Use average="macro" F1 when classes are imbalanced.

What is a good first model for a new dataset?

Logistic regression or Ridge for baselines.
RandomForest for a non-linear check without heavy tuning.
Compare both before investing in gradient boosting.

How do I interpret feature importance?

for name, imp in zip(feature_names, reg.feature_importances_):
    print(f"{name}: {imp:.3f}")

Tree models expose feature_importances_.
Linear models use coefficient magnitudes (after scaling).

Should I always use ensemble methods?

Ensembles win on tabular Kaggle-style data.
Linear models win when interpretability and speed matter.
Start simple, upgrade when validation justifies complexity.

How does sklearn handle string labels?

Most classifiers accept string y labels directly.
Internally encoded; predictions return the original label type.

When do I need polynomial features?

When linear models underfit but relationships are smooth.
Increases feature count rapidly - pair with regularization.
Tree models usually make polynomials unnecessary.

Model Evaluation - choosing the right metrics
Feature Engineering - preprocessing per algorithm
Gradient Boosting - XGBoost and LightGBM
Hyperparameter Tuning - optimizing model params
Imbalanced Data - class-weight and resampling

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Recipe

Quick-reference recipe card - copy-paste ready.

from sklearn.linear_model import Ridge, LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingRegressor
 
# Classification
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
 
# Regression
reg = Ridge(alpha=1.0)
reg.fit(X_train, y_train)

When to reach for this:

Predicting a numeric outcome (price, demand, latency) - regression.
Assigning categories (churn, fraud, sentiment) - classification.
Tabular data with labeled training examples.
Baseline modeling before deep learning or gradient boosting tuning.

Working Example

"""supervised_learning.py - regression and classification with sklearn."""
from __future__ import annotations
 
import pandas as pd
from sklearn.datasets import fetch_california_housing, load_breast_cancer
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.metrics import mean_absolute_error, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
 
# --- Classification ---
cancer = load_breast_cancer(as_frame=True)
Xc, yc = cancer.data, cancer.target
Xc_train, Xc_test, yc_train, yc_test = train_test_split(Xc, yc, test_size=0.2, random_state=42)
 
clf_pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000, random_state=42)),
])
clf_pipe.fit(Xc_train, yc_train)
proba = clf_pipe.predict_proba(Xc_test)[:, 1]
print("classification AUC:", roc_auc_score(yc_test, proba))
 
# --- Regression ---
housing = fetch_california_housing(as_frame=True)
Xr, yr = housing.data, housing.target
Xr_train, Xr_test, yr_train, yr_test = train_test_split(Xr, yr, test_size=0.2, random_state=42)
 
reg = RandomForestRegressor(n_estimators=200, random_state=42, n_jobs=-1)
reg.fit(Xr_train, yr_train)
print("regression MAE:", mean_absolute_error(yr_test, reg.predict(Xr_test)))

What this demonstrates:

Same fit/predict API for regression and classification.
Scaling before logistic regression; tree models skip scaling.
Probability outputs for classification ranking metrics.
MAE as an interpretable regression metric (dollars, units).

Deep Dive

How It Works

Training minimizes a loss function over labeled examples (log loss, MSE, hinge loss).
Linear models learn weighted sums of features; fast and interpretable.
Tree ensembles partition feature space with axis-aligned splits; capture non-linear interactions.
Regularization (L1/L2) penalizes large weights to reduce overfitting.
Decision boundary complexity grows with model capacity - balance with validation metrics.

Algorithm Selection

Problem	Start With	Upgrade To
Binary classification	LogisticRegression	RandomForest, XGBoost
Multi-class	LogisticRegression (multinomial)	RandomForestClassifier
Regression (linear-ish)	Ridge	GradientBoostingRegressor
Regression (non-linear)	RandomForestRegressor	LightGBM
Small data, interpretability	Linear models	-
Large tabular, accuracy	Gradient boosting	-

Python Notes

from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
 
# SVM: strong with scaled features, slower on large data
svm = Pipeline([("scale", StandardScaler()), ("clf", SVC(probability=True))])
 
# k-NN: lazy learner, sensitive to scale and dimensionality
knn = Pipeline([("scale", StandardScaler()), ("clf", KNeighborsClassifier(n_neighbors=5))])

Gotchas

Using classification metrics for regression - accuracy on continuous targets is meaningless. Fix: MAE, RMSE, R-squared for regression.
Not scaling for SVM/k-NN/logistic regression - distance and gradient methods fail on unscaled features. Fix: StandardScaler in a pipeline.
Ignoring class imbalance - accuracy looks good while the model predicts the majority class. Fix: F1, ROC-AUC, or resampling.
Overfitting with deep trees - max_depth=None memorizes noise on small data. Fix: limit depth, increase min_samples_leaf, use CV.
Multicollinearity in linear regression - unstable coefficients with correlated features. Fix: Ridge regression or feature selection.
Predicting without predict_proba - some models need probability=True (SVC) for probability outputs. Fix: check estimator support before using proba.

Alternatives

Alternative	Use When	Don't Use When
Linear models	Interpretability, fast baseline	Complex non-linear interactions dominate
Random Forest	Robust default, little tuning	Need maximum accuracy on large tabular
Gradient boosting	Best tabular accuracy	Need fast training iteration or interpretability
Deep learning	Images, text, sequences	Small tabular with <10k rows

FAQs

What is the difference between regression and classification?

Regression outputs continuous values (price = 249.99).
Classification outputs discrete labels (spam = yes/no).
Multi-class classification has 3+ categories.

When should I use logistic regression?

Strong linear baseline for classification.
Interpretable coefficients for stakeholder review.
Fast training on high-dimensional sparse text features.

What does regularization alpha do in Ridge?

Higher alpha shrinks coefficients toward zero, reducing overfitting.
alpha=0 is ordinary least squares.
Tune with cross-validation (RidgeCV).

How do tree models handle missing values?

sklearn RandomForest does not natively handle NaN - impute first.
XGBoost and LightGBM learn optimal missing-value directions.
Impute inside a pipeline for sklearn trees.

What is the bias-variance tradeoff?

Simple models underfit (high bias); complex models overfit (high variance).
Validation metrics reveal the sweet spot.
Regularization and ensembling reduce variance.

Can I use the same pipeline for train and production?

Yes - joblib.dump the fitted pipeline.
Production calls pipe.predict(new_row) with the same column names.

How do I handle multi-class classification?

LogisticRegression(multi_class="multinomial", max_iter=1000)
# or
RandomForestClassifier()  # handles multi-class natively

Use average="macro" F1 when classes are imbalanced.

What is a good first model for a new dataset?

Logistic regression or Ridge for baselines.
RandomForest for a non-linear check without heavy tuning.
Compare both before investing in gradient boosting.

How do I interpret feature importance?

for name, imp in zip(feature_names, reg.feature_importances_):
    print(f"{name}: {imp:.3f}")

Tree models expose feature_importances_.
Linear models use coefficient magnitudes (after scaling).

Should I always use ensemble methods?

Ensembles win on tabular Kaggle-style data.
Linear models win when interpretability and speed matter.
Start simple, upgrade when validation justifies complexity.

How does sklearn handle string labels?

Most classifiers accept string y labels directly.
Internally encoded; predictions return the original label type.

When do I need polynomial features?

When linear models underfit but relationships are smooth.
Increases feature count rapidly - pair with regularization.
Tree models usually make polynomials unnecessary.

Model Evaluation - choosing the right metrics
Feature Engineering - preprocessing per algorithm
Gradient Boosting - XGBoost and LightGBM
Hyperparameter Tuning - optimizing model params
Imbalanced Data - class-weight and resampling

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Supervised Learning

Search Documentation

Supervised Learning