Generators & yield
Generator functions use yield to produce a stream of values lazily. They preserve local state between steps - ideal for large files, infinite sequences, and ETL pipelines.
Recipe
def squares(n: int):
for i in range(n):
yield i * i
for value in squares(5):
print(value)When to reach for this:
- Reading large files line by line
- Producing unbounded sequences (IDs, retries)
- Chaining map/filter stages without lists
- Implementing cooperative multitasking primitives
Working Example
from pathlib import Path
def read_lines(path: Path):
with path.open(encoding="utf-8") as fh:
for line in fh:
yield line.rstrip("\n")
def filter_nonempty(lines):
for line in lines:
if line.strip():
yield line
def parse_csv_rows(lines):
for line in lines:
yield line.split(",")
def pipeline(path: Path):
lines = read_lines(path)
nonempty = filter_nonempty(lines)
return parse_csv_rows(nonempty)
if __name__ == "__main__":
sample = Path("data.csv")
sample.write_text("a,b\n\n c,d\n", encoding="utf-8")
for row in pipeline(sample):
print(row)What this demonstrates:
- Each stage is lazy - no full file in memory
- Generators compose by passing iterator to next function
- File handle closed when generator garbage-collected or exhausted (with context in gen)
- Pipeline returns generator - consumer drives execution
Deep Dive
How It Works
- Generator object - Created when generator function called - body not run yet.
- yield suspends - Saves frame locals; resumes on
next(). - StopIteration - Raised when generator returns or falls off end.
- yield from - Delegates to sub-iterator (see dedicated article).
- Generator state -
.gi_frameholds execution point (debugging).
Memory Profile
| Approach | Memory |
|---|---|
list(process()) | O(n) all items |
| generator pipeline | O(1) per stage state |
Python Notes
# generator expression
total = sum(x * x for x in range(1_000_000))
# close generator early
gen = squares(100)
next(gen)
gen.close() # raises GeneratorExit insideGotchas
- Exhausted generator - Second loop does nothing. Fix: Recreate or materialize once.
- Resource leak without with in generator - If consumer stops early, file may stay open until GC. Fix:
contextlib.closingor try/finally in generator. - Mutable default arg in generator - Same as functions. Fix: None sentinel pattern.
- Side effects order - Lazy means effects happen on consumption, not definition. Fix: Document lazy semantics.
- Debugging stack traces - Suspend frames confuse newcomers - use
gen.send(None)clarity in docs.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| List comprehension | Small n, need reuse | Large streams |
itertools | Standard iterator ops | Custom domain logic per item |
| async generator | Async I/O streams | Sync file reads |
| pandas chunk iterator | Tabular batches | Simple line parsing |
FAQs
generator vs list?
Generator lazy single-pass. List stores all elements for reuse and indexing.
What is yield from?
Delegates iteration to another iterable - flattens nested generators efficiently.
Can generators return values?
return value sets StopIteration.value (PEP 380) - consumed via yield from, rarely needed alone.
send() purpose?
Coroutine-style communication into generator - advanced; asyncio uses similar ideas.
How stop iteration early?
gen.close() triggers GeneratorExit at yield point - clean up resources in try/finally in gen.
generator throw?
Inject exception at yield point - testing and coroutine patterns.
typing Generator?
Generator[YieldType, SendType, ReturnType] for advanced typing - often Iterator[T] enough.
multiple yield in loop?
Normal pattern - each yield produces one consumer step.
performance vs list?
Generators save memory; list may be faster for tiny sequences due to overhead.
tee iterator?
itertools.tee splits one iterator into two buffered iterators - memory cost grows.
Related
- yield from & Sub-generators - delegation
- Iterators & the Iterator Protocol - protocol basics
- Context Managers - cleanup in generators
- itertools - chain, islice
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.