Supervised Learning
Supervised learning maps input features to known labels. Regression predicts continuous values; classification assigns discrete categories. sklearn provides a consistent API across algorithms.
Recipe
Quick-reference recipe card - copy-paste ready.
from sklearn.linear_model import Ridge, LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingRegressor
# Classification
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
# Regression
reg = Ridge(alpha=1.0)
reg.fit(X_train, y_train)When to reach for this:
- Predicting a numeric outcome (price, demand, latency) - regression.
- Assigning categories (churn, fraud, sentiment) - classification.
- Tabular data with labeled training examples.
- Baseline modeling before deep learning or gradient boosting tuning.
Working Example
"""supervised_learning.py - regression and classification with sklearn."""
from __future__ import annotations
import pandas as pd
from sklearn.datasets import fetch_california_housing, load_breast_cancer
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.metrics import mean_absolute_error, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
# --- Classification ---
cancer = load_breast_cancer(as_frame=True)
Xc, yc = cancer.data, cancer.target
Xc_train, Xc_test, yc_train, yc_test = train_test_split(Xc, yc, test_size=0.2, random_state=42)
clf_pipe = Pipeline([
("scale", StandardScaler()),
("clf", LogisticRegression(max_iter=1000, random_state=42)),
])
clf_pipe.fit(Xc_train, yc_train)
proba = clf_pipe.predict_proba(Xc_test)[:, 1]
print("classification AUC:", roc_auc_score(yc_test, proba))
# --- Regression ---
housing = fetch_california_housing(as_frame=True)
Xr, yr = housing.data, housing.target
Xr_train, Xr_test, yr_train, yr_test = train_test_split(Xr, yr, test_size=0.2, random_state=42)
reg = RandomForestRegressor(n_estimators=200, random_state=42, n_jobs=-1)
reg.fit(Xr_train, yr_train)
print("regression MAE:", mean_absolute_error(yr_test, reg.predict(Xr_test)))What this demonstrates:
- Same
fit/predictAPI for regression and classification. - Scaling before logistic regression; tree models skip scaling.
- Probability outputs for classification ranking metrics.
- MAE as an interpretable regression metric (dollars, units).
Deep Dive
How It Works
- Training minimizes a loss function over labeled examples (log loss, MSE, hinge loss).
- Linear models learn weighted sums of features; fast and interpretable.
- Tree ensembles partition feature space with axis-aligned splits; capture non-linear interactions.
- Regularization (L1/L2) penalizes large weights to reduce overfitting.
- Decision boundary complexity grows with model capacity - balance with validation metrics.
Algorithm Selection
| Problem | Start With | Upgrade To |
|---|---|---|
| Binary classification | LogisticRegression | RandomForest, XGBoost |
| Multi-class | LogisticRegression (multinomial) | RandomForestClassifier |
| Regression (linear-ish) | Ridge | GradientBoostingRegressor |
| Regression (non-linear) | RandomForestRegressor | LightGBM |
| Small data, interpretability | Linear models | - |
| Large tabular, accuracy | Gradient boosting | - |
Python Notes
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
# SVM: strong with scaled features, slower on large data
svm = Pipeline([("scale", StandardScaler()), ("clf", SVC(probability=True))])
# k-NN: lazy learner, sensitive to scale and dimensionality
knn = Pipeline([("scale", StandardScaler()), ("clf", KNeighborsClassifier(n_neighbors=5))])Gotchas
- Using classification metrics for regression - accuracy on continuous targets is meaningless. Fix: MAE, RMSE, R-squared for regression.
- Not scaling for SVM/k-NN/logistic regression - distance and gradient methods fail on unscaled features. Fix:
StandardScalerin a pipeline. - Ignoring class imbalance - accuracy looks good while the model predicts the majority class. Fix: F1, ROC-AUC, or resampling.
- Overfitting with deep trees -
max_depth=Nonememorizes noise on small data. Fix: limit depth, increasemin_samples_leaf, use CV. - Multicollinearity in linear regression - unstable coefficients with correlated features. Fix: Ridge regression or feature selection.
- Predicting without
predict_proba- some models needprobability=True(SVC) for probability outputs. Fix: check estimator support before using proba.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Linear models | Interpretability, fast baseline | Complex non-linear interactions dominate |
| Random Forest | Robust default, little tuning | Need maximum accuracy on large tabular |
| Gradient boosting | Best tabular accuracy | Need fast training iteration or interpretability |
| Deep learning | Images, text, sequences | Small tabular with <10k rows |
FAQs
What is the difference between regression and classification?
- Regression outputs continuous values (price = 249.99).
- Classification outputs discrete labels (spam = yes/no).
- Multi-class classification has 3+ categories.
When should I use logistic regression?
- Strong linear baseline for classification.
- Interpretable coefficients for stakeholder review.
- Fast training on high-dimensional sparse text features.
What does regularization alpha do in Ridge?
- Higher
alphashrinks coefficients toward zero, reducing overfitting. alpha=0is ordinary least squares.- Tune with cross-validation (
RidgeCV).
How do tree models handle missing values?
- sklearn
RandomForestdoes not natively handle NaN - impute first. - XGBoost and LightGBM learn optimal missing-value directions.
- Impute inside a pipeline for sklearn trees.
What is the bias-variance tradeoff?
- Simple models underfit (high bias); complex models overfit (high variance).
- Validation metrics reveal the sweet spot.
- Regularization and ensembling reduce variance.
Can I use the same pipeline for train and production?
- Yes -
joblib.dumpthe fitted pipeline. - Production calls
pipe.predict(new_row)with the same column names.
How do I handle multi-class classification?
LogisticRegression(multi_class="multinomial", max_iter=1000)
# or
RandomForestClassifier() # handles multi-class natively- Use
average="macro"F1 when classes are imbalanced.
What is a good first model for a new dataset?
- Logistic regression or Ridge for baselines.
- RandomForest for a non-linear check without heavy tuning.
- Compare both before investing in gradient boosting.
How do I interpret feature importance?
for name, imp in zip(feature_names, reg.feature_importances_):
print(f"{name}: {imp:.3f}")- Tree models expose
feature_importances_. - Linear models use coefficient magnitudes (after scaling).
Should I always use ensemble methods?
- Ensembles win on tabular Kaggle-style data.
- Linear models win when interpretability and speed matter.
- Start simple, upgrade when validation justifies complexity.
How does sklearn handle string labels?
- Most classifiers accept string y labels directly.
- Internally encoded; predictions return the original label type.
When do I need polynomial features?
- When linear models underfit but relationships are smooth.
- Increases feature count rapidly - pair with regularization.
- Tree models usually make polynomials unnecessary.
Related
- Model Evaluation - choosing the right metrics
- Feature Engineering - preprocessing per algorithm
- Gradient Boosting - XGBoost and LightGBM
- Hyperparameter Tuning - optimizing model params
- Imbalanced Data - class-weight and resampling
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.