GroupBy & Aggregation

pandas groupby implements split-apply-combine: partition rows by keys, run a function per group, and stitch results into a summary or transformed frame.

Recipe

Quick-reference recipe card - copy-paste ready.

import pandas as pd
 
summary = (
    df.groupby("region", observed=True)["revenue"]
    .agg(total="sum", avg="mean", n="count")
    .reset_index()
)
pivot = pd.pivot_table(df, index="month", columns="region", values="revenue", aggfunc="sum")

When to reach for this:

Regional or product rollups on fact tables
Windowed metrics that need per-group statistics
Pivoting long data to wide reporting layouts
Feature engineering (group-level means for ML)

Working Example

import pandas as pd
 
sales = pd.DataFrame(
    {
        "region": pd.Categorical(["East", "West", "East", "West", "East"]),
        "month": ["2025-01", "2025-01", "2025-02", "2025-02", "2025-02"],
        "revenue": [120.0, 340.0, 150.0, 280.0, 130.0],
        "units": [10, 25, 12, 20, 11],
    }
)
 
# Multi-metric aggregation
by_region = (
    sales.groupby("region", observed=True)
    .agg(revenue_total=("revenue", "sum"), units_avg=("units", "mean"))
    .reset_index()
)
 
# Per-row share of regional total (transform keeps row count)
sales = sales.copy()
sales["share"] = sales.groupby("region", observed=True)["revenue"].transform(
    lambda s: s / s.sum()
)
 
# Pivot for reporting
pivot = pd.pivot_table(
    sales,
    index="month",
    columns="region",
    values="revenue",
    aggfunc="sum",
    fill_value=0,
    observed=True,
)
 
print(by_region)
print(sales[["region", "revenue", "share"]])
print(pivot)

What this demonstrates:

Named aggregation with .agg(col=(field, func))
transform for row-level group fractions
pivot_table with fill_value for sparse months
observed=True on categorical keys

Deep Dive

How It Works

Split: hash or sort rows by group keys.
Apply: run reduction (sum, mean) or transform (rank, fillna).
Combine: stack results into Series/DataFrame with a MultiIndex or flat columns.
as_index=False (pandas 2.x default in many APIs) returns keys as columns.

agg vs transform vs apply

Method	Output rows	Use
`agg`	One per group	Summaries
`transform`	Same as input	Per-row group stats
`apply`	Varies	Custom Python logic (slower)

Python Notes

import pandas as pd
 
# Multiple keys
df.groupby(["region", "month"], observed=True)["revenue"].sum()
 
# Filter groups by aggregate condition
df.groupby("region").filter(lambda g: g["revenue"].sum() > 500)

Gotchas

Including NA in group keys - dropna=False keeps NaN groups; often accidental. Fix: dropna=True default or clean keys first.
Unobserved categorical levels - empty regions appear without observed=True. Fix: pass observed=True in pandas 2.x groupby/pivot.
apply returning inconsistent shapes - breaks combine step. Fix: prefer built-in agg/transform; vectorize custom logic.
Pivot duplicate keys - multiple rows per (month, region) need aggfunc. Fix: choose sum, mean, or first explicitly.
Sorting groupby output - unsorted index makes reports hard to read. Fix: .sort_values("revenue_total", ascending=False).

Alternatives

Alternative	Use When	Don't Use When
Polars `group_by`	Faster lazy aggregations on big data	Small frames where pandas is already loaded
DuckDB SQL	Complex joins + aggregates in one query	Simple one-table rollups
`pd.crosstab`	Frequency counts of two factors	Numeric revenue sums
NumPy `bincount`	Integer bucket sums on 1-D data	Multiple columns and NA handling

FAQs

What is observed=True?

Limits categorical groupby to categories present in data.
Prevents empty groups from unused category levels.

How do I count distinct values per group?

df.groupby("region")["customer_id"].nunique()

How do I get rank within group?

df["rank"] = df.groupby("region")["revenue"].rank(ascending=False)

Can I aggregate multiple columns differently?

df.groupby("region").agg({"revenue": "sum", "units": "max"})

Why is groupby slow?

Python apply per group is the usual culprit.
Use vectorized agg, Polars, or DuckDB for millions of rows.

How do I reset index after groupby?

out = df.groupby("region").sum().reset_index()

What is named aggregation?

.agg(total=("revenue", "sum")) names output columns directly.
Clearer than MultiIndex columns from .agg(["sum", "mean"]).

How do pivot_table and groupby differ?

Same engine underneath - pivot is wide layout sugar.
Use pivot when consumers expect columns per category.

How do I handle time-based groups?

df.groupby(pd.Grouper(key="date", freq="ME")) for month-end buckets.
Ensure datetime dtype and timezone consistency first.

Can I groupby and plot in one line?

df.groupby("region")["revenue"].sum().plot(kind="bar")

Sort values first for readable charts.

pandas Series & DataFrames - indexing
Joins & Merges - enrich before grouping
Time Series - Grouper and resample
Polars - fast group_by at scale

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

GroupBy & Aggregation

pandas groupby implements split-apply-combine: partition rows by keys, run a function per group, and stitch results into a summary or transformed frame.

Recipe

Quick-reference recipe card - copy-paste ready.

import pandas as pd
 
summary = (
    df.groupby("region", observed=True)["revenue"]
    .agg(total="sum", avg="mean", n="count")
    .reset_index()
)
pivot = pd.pivot_table(df, index="month", columns="region", values="revenue", aggfunc="sum")

When to reach for this:

Regional or product rollups on fact tables
Windowed metrics that need per-group statistics
Pivoting long data to wide reporting layouts
Feature engineering (group-level means for ML)

Working Example

import pandas as pd
 
sales = pd.DataFrame(
    {
        "region": pd.Categorical(["East", "West", "East", "West", "East"]),
        "month": ["2025-01", "2025-01", "2025-02", "2025-02", "2025-02"],
        "revenue": [120.0, 340.0, 150.0, 280.0, 130.0],
        "units": [10, 25, 12, 20, 11],
    }
)
 
# Multi-metric aggregation
by_region = (
    sales.groupby("region", observed=True)
    .agg(revenue_total=("revenue", "sum"), units_avg=("units", "mean"))
    .reset_index()
)
 
# Per-row share of regional total (transform keeps row count)
sales = sales.copy()
sales["share"] = sales.groupby("region", observed=True)["revenue"].transform(
    lambda s: s / s.sum()
)
 
# Pivot for reporting
pivot = pd.pivot_table(
    sales,
    index="month",
    columns="region",
    values="revenue",
    aggfunc="sum",
    fill_value=0,
    observed=True,
)
 
print(by_region)
print(sales[["region", "revenue", "share"]])
print(pivot)

What this demonstrates:

Named aggregation with .agg(col=(field, func))
transform for row-level group fractions
pivot_table with fill_value for sparse months
observed=True on categorical keys

Deep Dive

How It Works

Split: hash or sort rows by group keys.
Apply: run reduction (sum, mean) or transform (rank, fillna).
Combine: stack results into Series/DataFrame with a MultiIndex or flat columns.
as_index=False (pandas 2.x default in many APIs) returns keys as columns.

agg vs transform vs apply

Method	Output rows	Use
`agg`	One per group	Summaries
`transform`	Same as input	Per-row group stats
`apply`	Varies	Custom Python logic (slower)

Python Notes

import pandas as pd
 
# Multiple keys
df.groupby(["region", "month"], observed=True)["revenue"].sum()
 
# Filter groups by aggregate condition
df.groupby("region").filter(lambda g: g["revenue"].sum() > 500)

Gotchas

Including NA in group keys - dropna=False keeps NaN groups; often accidental. Fix: dropna=True default or clean keys first.
Unobserved categorical levels - empty regions appear without observed=True. Fix: pass observed=True in pandas 2.x groupby/pivot.
apply returning inconsistent shapes - breaks combine step. Fix: prefer built-in agg/transform; vectorize custom logic.
Pivot duplicate keys - multiple rows per (month, region) need aggfunc. Fix: choose sum, mean, or first explicitly.
Sorting groupby output - unsorted index makes reports hard to read. Fix: .sort_values("revenue_total", ascending=False).

Alternatives

Alternative	Use When	Don't Use When
Polars `group_by`	Faster lazy aggregations on big data	Small frames where pandas is already loaded
DuckDB SQL	Complex joins + aggregates in one query	Simple one-table rollups
`pd.crosstab`	Frequency counts of two factors	Numeric revenue sums
NumPy `bincount`	Integer bucket sums on 1-D data	Multiple columns and NA handling

FAQs

What is observed=True?

Limits categorical groupby to categories present in data.
Prevents empty groups from unused category levels.

How do I count distinct values per group?

df.groupby("region")["customer_id"].nunique()

How do I get rank within group?

df["rank"] = df.groupby("region")["revenue"].rank(ascending=False)

Can I aggregate multiple columns differently?

df.groupby("region").agg({"revenue": "sum", "units": "max"})

Why is groupby slow?

Python apply per group is the usual culprit.
Use vectorized agg, Polars, or DuckDB for millions of rows.

How do I reset index after groupby?

out = df.groupby("region").sum().reset_index()

What is named aggregation?

.agg(total=("revenue", "sum")) names output columns directly.
Clearer than MultiIndex columns from .agg(["sum", "mean"]).

How do pivot_table and groupby differ?

Same engine underneath - pivot is wide layout sugar.
Use pivot when consumers expect columns per category.

How do I handle time-based groups?

df.groupby(pd.Grouper(key="date", freq="ME")) for month-end buckets.
Ensure datetime dtype and timezone consistency first.

Can I groupby and plot in one line?

df.groupby("region")["revenue"].sum().plot(kind="bar")

Sort values first for readable charts.

pandas Series & DataFrames - indexing
Joins & Merges - enrich before grouping
Time Series - Grouper and resample
Polars - fast group_by at scale

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

GroupBy & Aggregation

Search Documentation

GroupBy & Aggregation