Comprehensions & Generator Expressions
Comprehensions build lists, dicts, and sets from iterables in a single expression. Generator expressions look similar but return a lazy iterator - ideal for pipelines that never need the full collection in memory.
Recipe
squares = [n * n for n in range(10)]
evens = {n for n in range(20) if n % 2 == 0}
index = {name: i for i, name in enumerate(["ada", "linus"])}
total = sum(x * x for x in range(100_000)) # generator - no big listWhen to reach for this:
- Transforming or filtering sequences declaratively
- Building lookup dicts from parallel iterables
- Streaming large datasets through sum/any/all/max
- Replacing three-line append loops with one expression
Working Example
from pathlib import Path
def load_word_counts(paths: list[Path]) -> dict[str, int]:
words = (
word.lower()
for path in paths
for line in path.read_text(encoding="utf-8").splitlines()
for word in line.split()
)
return {word: len(word) for word in sorted(set(words))}
def active_user_emails(rows: list[dict[str, object]]) -> list[str]:
return [
str(row["email"])
for row in rows
if row.get("active") and "email" in row
]
def chunk_indices(size: int, chunk: int) -> list[tuple[int, int]]:
return [(i, min(i + chunk, size)) for i in range(0, size, chunk)]
if __name__ == "__main__":
print(active_user_emails([
{"email": "a@x.com", "active": True},
{"email": "b@x.com", "active": False},
]))
print(chunk_indices(10, 3))What this demonstrates:
- Nested generator expression flattens files without intermediate lists
- List comprehension filters and projects dict rows
- Set comprehension inside dict comp deduplicates words before length mapping
- Generator argument to
sumavoids allocating 100k integers
Deep Dive
How It Works
- Eager comprehensions - List/dict/set comps construct the full collection immediately.
- Generator expressions - Parenthesized
(x for x in it)return a generator object. - Single expression - No statements inside comps - only expressions and
iffilters. - Scope - Comprehensions have their own local scope in Python 3 (loop variables do not leak).
- Chaining - Multiple
forclauses nest left-to-right like nested loops.
When to Pick Which
| Form | Memory | Reuse |
|---|---|---|
[...] list comp | Stores all items | Yes - random access |
{...} set comp | Stores unique items | Yes |
{k: v ...} dict comp | Stores all pairs | Yes |
(...) generator | O(1) iterator state | Consume once |
Python Notes
# Walrus in comprehension (3.8+)
filtered = [y for x in data if (y := transform(x)) is not None]
# dict.fromkeys for simple key initialization
unique_order = list(dict.fromkeys(items)) # preserve order, dedupeGotchas
- Side effects in comps -
[print(x) for x in items]builds a list of None. Fix: Use a plainforloop for effects. - Huge list comp -
[process(x) for x in billion_rows]exhausts RAM. Fix: Generator and stream processing. - Unreadable nesting - Three
forclauses on one line obscure intent. Fix: Break into helper functions or useitertools. - Dict comp key collision - Later keys overwrite earlier silently. Fix: Validate uniqueness or use a loop with checks.
- Exhausted generator - Iterating twice yields nothing the second time. Fix: Materialize to list or recreate generator.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Generator expression | Large or infinite streams | Need indexing or len() |
for loop | Side effects, complex branching | Simple map/filter |
map/filter | Functional style with functions | Multiple if/for clauses |
itertools | Sliding windows, chaining | Simple one-liner comp |
FAQs
Can I use else in a comprehension?
Yes: [x if x > 0 else 0 for x in nums] - expression before for, not the same as loop else.
Are comprehensions faster than loops?
Often yes in CPython - optimized bytecode and no repeated method lookup for append.
When must I use a generator?
When data may not fit in memory or you only pass through once to aggregate functions.
Do comprehension variables leak?
No in Python 3 - x in [x for x in items] is local to the comprehension.
How do I flatten a nested list?
[y for row in matrix for y in row] - multiple for clauses flatten one level.
Can I build a dict from two lists?
{k: v for k, v in zip(keys, values)} - or dict(zip(keys, values)) when no filter needed.
What is a set comprehension?
{expr for x in items} - curly braces without colon. Produces a set, not dict.
Should I assign a generator to a variable?
Yes when reusing the pipeline definition: gen = (x for x in data if pred(x)) then pass to functions.
How do I debug a complex comp?
Expand to a for loop temporarily, or extract the inner expression to a named function.
Is list(map(...)) dead?
Not always - map with an existing function can be clearer. Comps win for inline transforms with filters.
Related
- Generators & yield - full generator functions
- itertools - advanced iterator composition
- Lists & Tuples - list operations
- Control Flow - when loops beat comps
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.