Feature Engineering

Feature engineering transforms raw columns into representations models can learn from. Encoding categoricals, scaling numerics, and composing transforms with ColumnTransformer are the core patterns for tabular ML.

Recipe

Quick-reference recipe card - copy-paste ready.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
 
preprocessor = ColumnTransformer([
    ("num", StandardScaler(), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])
pipe = Pipeline([("prep", preprocessor), ("clf", RandomForestClassifier())])

When to reach for this:

• Mixed numeric and categorical columns in the same dataset.
• Models sensitive to feature scale (SVM, logistic regression, neural nets on tabular data).
• High-cardinality categoricals that need hashing or target encoding.
• Reproducible preprocessing bundled with the model for deployment.

Working Example

"""feature_engineering.py - mixed-type preprocessing with ColumnTransformer."""
from __future__ import annotations
 
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
 
df = pd.DataFrame({
    "age": [25, 34, 45, 29, 52],
    "income": [45000, 62000, 88000, 51000, 95000],
    "city": ["nyc", "la", "nyc", "chicago", "la"],
    "plan": ["basic", "pro", "pro", "basic", "enterprise"],
    "churned": [0, 0, 1, 0, 1],
})
 
X = df.drop(columns=["churned"])
y = df["churned"]
numeric_cols = ["age", "income"]
categorical_cols = ["city", "plan"]
 
preprocessor = ColumnTransformer(
    transformers=[
        ("num", StandardScaler(), numeric_cols),
        ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), categorical_cols),
    ],
    remainder="drop",
)
 
pipe = Pipeline([
    ("prep", preprocessor),
    ("clf", GradientBoostingClassifier(random_state=42)),
])
 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)
pipe.fit(X_train, y_train)
print(classification_report(y_test, pipe.predict(X_test)))

What this demonstrates:

• Different transforms applied to numeric and categorical column groups.
• OneHotEncoder with handle_unknown="ignore" for production safety.
• sparse_output=False for dense output compatible with tree and linear models.
• Full pipeline serialization - preprocessing travels with the classifier.

Deep Dive

How It Works

• Scaling centers and scales numeric columns so gradient-based models converge faster.
• One-hot encoding expands categoricals into binary indicator columns per category.
• ColumnTransformer runs each sub-transformer on its assigned columns and concatenates results.
• Pipelines call fit on training data per CV fold, preventing statistic leakage.
• Feature names can be recovered with get_feature_names_out() after fitting.

Common Transforms

Python Notes

from sklearn.preprocessing import FunctionTransformer
import numpy as np
 
log_transform = FunctionTransformer(np.log1p, validate=False)
# Chain imputation before scaling in the numeric branch:
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline as SkPipeline
 
numeric_pipe = SkPipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])

Gotchas

• Fitting on full data - computing mean/std or category lists from test rows leaks information. Fix: fit inside a pipeline on training folds only.
• Unseen categories at inference - new category values crash a plain OneHotEncoder. Fix: handle_unknown="ignore" or handle_unknown="infrequent_if_exist".
• Target encoding leakage - encoding categories with the target mean using all data inflates CV scores. Fix: use TargetEncoder inside a pipeline with proper CV.
• Scaling tree models unnecessarily - random forests and gradient boosting are scale-invariant. Fix: skip scaling for tree ensembles; still encode categoricals.
• High-cardinality one-hot - thousands of dummy columns slow training and overfit. Fix: hashing (FeatureHasher), frequency encoding, or embeddings.
• Polynomial explosion - PolynomialFeatures(degree=3) on 20 columns creates thousands of features. Fix: limit degree, use feature selection, or interaction-only mode.

Alternatives

FAQs

pipe.fit(X_train, y_train)
names = pipe.named_steps["prep"].get_feature_names_out()
print(list(names))

from sklearn.impute import SimpleImputer
num_pipe = Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())])

df["order_year"] = df["order_date"].dt.year
df["order_dow"] = df["order_date"].dt.dayofweek

Xt = preprocessor.fit_transform(X_train)
print(Xt.shape)  # rows x total encoded features

Related

• Pipelines - chaining transforms and models
• Datasets & Splits - fit transforms on train only
• Supervised Learning - model-specific preprocessing needs
• Imbalanced Data - resampling inside pipelines
• Gradient Boosting - tree models and categoricals

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.