Generators & yield

Q: Can generators return values?

return value sets StopIteration.value (PEP 380) - consumed via yield from, rarely needed alone.

Q: How stop iteration early?

gen.close() triggers GeneratorExit at yield point - clean up resources in try/finally in gen.

Q: typing Generator?

Generator[YieldType, SendType, ReturnType] for advanced typing - often Iterator[T] enough.

Q: tee iterator?

itertools.tee splits one iterator into two buffered iterators - memory cost grows.

Generator functions use yield to produce a stream of values lazily. They preserve local state between steps - ideal for large files, infinite sequences, and ETL pipelines.

Recipe

def squares(n: int):
    for i in range(n):
        yield i * i
 
for value in squares(5):
    print(value)

When to reach for this:

Reading large files line by line
Producing unbounded sequences (IDs, retries)
Chaining map/filter stages without lists
Implementing cooperative multitasking primitives

Working Example

from pathlib import Path
 
def read_lines(path: Path):
    with path.open(encoding="utf-8") as fh:
        for line in fh:
            yield line.rstrip("\n")
 
def filter_nonempty(lines):
    for line in lines:
        if line.strip():
            yield line
 
def parse_csv_rows(lines):
    for line in lines:
        yield line.split(",")
 
def pipeline(path: Path):
    lines = read_lines(path)
    nonempty = filter_nonempty(lines)
    return parse_csv_rows(nonempty)
 
if __name__ == "__main__":
    sample = Path("data.csv")
    sample.write_text("a,b\n\n c,d\n", encoding="utf-8")
    for row in pipeline(sample):
        print(row)

What this demonstrates:

Each stage is lazy - no full file in memory
Generators compose by passing iterator to next function
File handle closed when generator garbage-collected or exhausted (with context in gen)
Pipeline returns generator - consumer drives execution

Deep Dive

How It Works

Generator object - Created when generator function called - body not run yet.
yield suspends - Saves frame locals; resumes on next().
StopIteration - Raised when generator returns or falls off end.
yield from - Delegates to sub-iterator (see dedicated article).
Generator state - .gi_frame holds execution point (debugging).

Memory Profile

Approach	Memory
`list(process())`	O(n) all items
generator pipeline	O(1) per stage state

Python Notes

# generator expression
total = sum(x * x for x in range(1_000_000))
 
# close generator early
gen = squares(100)
next(gen)
gen.close()  # raises GeneratorExit inside

Gotchas

Exhausted generator - Second loop does nothing. Fix: Recreate or materialize once.
Resource leak without with in generator - If consumer stops early, file may stay open until GC. Fix: contextlib.closing or try/finally in generator.
Mutable default arg in generator - Same as functions. Fix: None sentinel pattern.
Side effects order - Lazy means effects happen on consumption, not definition. Fix: Document lazy semantics.
Debugging stack traces - Suspend frames confuse newcomers - use gen.send(None) clarity in docs.

Alternatives

Alternative	Use When	Don't Use When
List comprehension	Small n, need reuse	Large streams
`itertools`	Standard iterator ops	Custom domain logic per item
async generator	Async I/O streams	Sync file reads
pandas chunk iterator	Tabular batches	Simple line parsing

FAQs

generator vs list?

Generator lazy single-pass. List stores all elements for reuse and indexing.

What is yield from?

Delegates iteration to another iterable - flattens nested generators efficiently.

Can generators return values?

return value sets StopIteration.value (PEP 380) - consumed via yield from, rarely needed alone.

send() purpose?

Coroutine-style communication into generator - advanced; asyncio uses similar ideas.

How stop iteration early?

gen.close() triggers GeneratorExit at yield point - clean up resources in try/finally in gen.

generator throw?

Inject exception at yield point - testing and coroutine patterns.

typing Generator?

Generator[YieldType, SendType, ReturnType] for advanced typing - often Iterator[T] enough.

multiple yield in loop?

Normal pattern - each yield produces one consumer step.

performance vs list?

Generators save memory; list may be faster for tiny sequences due to overhead.

tee iterator?

itertools.tee splits one iterator into two buffered iterators - memory cost grows.

yield from & Sub-generators - delegation
Iterators & the Iterator Protocol - protocol basics
Context Managers - cleanup in generators
itertools - chain, islice

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Recipe

def squares(n: int):
    for i in range(n):
        yield i * i
 
for value in squares(5):
    print(value)

When to reach for this:

Reading large files line by line
Producing unbounded sequences (IDs, retries)
Chaining map/filter stages without lists
Implementing cooperative multitasking primitives

Working Example

from pathlib import Path
 
def read_lines(path: Path):
    with path.open(encoding="utf-8") as fh:
        for line in fh:
            yield line.rstrip("\n")
 
def filter_nonempty(lines):
    for line in lines:
        if line.strip():
            yield line
 
def parse_csv_rows(lines):
    for line in lines:
        yield line.split(",")
 
def pipeline(path: Path):
    lines = read_lines(path)
    nonempty = filter_nonempty(lines)
    return parse_csv_rows(nonempty)
 
if __name__ == "__main__":
    sample = Path("data.csv")
    sample.write_text("a,b\n\n c,d\n", encoding="utf-8")
    for row in pipeline(sample):
        print(row)

What this demonstrates:

Each stage is lazy - no full file in memory
Generators compose by passing iterator to next function
File handle closed when generator garbage-collected or exhausted (with context in gen)
Pipeline returns generator - consumer drives execution

Deep Dive

How It Works

Generator object - Created when generator function called - body not run yet.
yield suspends - Saves frame locals; resumes on next().
StopIteration - Raised when generator returns or falls off end.
yield from - Delegates to sub-iterator (see dedicated article).
Generator state - .gi_frame holds execution point (debugging).

Memory Profile

Approach	Memory
`list(process())`	O(n) all items
generator pipeline	O(1) per stage state

Python Notes

# generator expression
total = sum(x * x for x in range(1_000_000))
 
# close generator early
gen = squares(100)
next(gen)
gen.close()  # raises GeneratorExit inside

Gotchas

Exhausted generator - Second loop does nothing. Fix: Recreate or materialize once.
Resource leak without with in generator - If consumer stops early, file may stay open until GC. Fix: contextlib.closing or try/finally in generator.
Mutable default arg in generator - Same as functions. Fix: None sentinel pattern.
Side effects order - Lazy means effects happen on consumption, not definition. Fix: Document lazy semantics.
Debugging stack traces - Suspend frames confuse newcomers - use gen.send(None) clarity in docs.

Alternatives

Alternative	Use When	Don't Use When
List comprehension	Small n, need reuse	Large streams
`itertools`	Standard iterator ops	Custom domain logic per item
async generator	Async I/O streams	Sync file reads
pandas chunk iterator	Tabular batches	Simple line parsing

FAQs

generator vs list?

Generator lazy single-pass. List stores all elements for reuse and indexing.

What is yield from?

Delegates iteration to another iterable - flattens nested generators efficiently.

Can generators return values?

return value sets StopIteration.value (PEP 380) - consumed via yield from, rarely needed alone.

send() purpose?

Coroutine-style communication into generator - advanced; asyncio uses similar ideas.

How stop iteration early?

gen.close() triggers GeneratorExit at yield point - clean up resources in try/finally in gen.

generator throw?

Inject exception at yield point - testing and coroutine patterns.

typing Generator?

Generator[YieldType, SendType, ReturnType] for advanced typing - often Iterator[T] enough.

multiple yield in loop?

Normal pattern - each yield produces one consumer step.

performance vs list?

Generators save memory; list may be faster for tiny sequences due to overhead.

tee iterator?

itertools.tee splits one iterator into two buffered iterators - memory cost grows.

yield from & Sub-generators - delegation
Iterators & the Iterator Protocol - protocol basics
Context Managers - cleanup in generators
itertools - chain, islice

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Generators & yield

Search Documentation

Generators & yield