Iterators & Decorators Best Practices

Q: testing lazy pipelines?

list(pipeline(data)) in tests; assert intermediate cleanup with mocks on close.

Guidelines for lazy iteration, decorator hygiene, and composable pipelines - the patterns that keep Python services memory-safe and debuggable.

How to Use This List

Review when building ETL pipelines or log processors.
Apply in decorator-heavy frameworks (auth, retry, metrics).
Pair with profiling when iterator chains grow deep.
Use in code review for lambda in loops and missing @wraps.

A - Iteration

Use generators or generator expressions for streams larger than memory. Materialize only at final consumer when needed.
Compose pipelines as small generator functions. Each stage does one transform - easier testing.
Prefer itertools for chain, islice, groupby, product. Battle-tested and lazy.
Sort before groupby when grouping by key. Consecutive-key requirement is easy to forget.
Document single-pass iterators. Callers must not assume replay without tee or list().

B - Generators and Resources

Use try/finally or @contextmanager inside generators holding files/sockets. Consumers may stop early.
Close generators with .close() in tests when cleanup matters. Triggers GeneratorExit at yield.
Avoid side effects in generator definition - effects run at consumption. Lazy semantics surprise newcomers.
Use yield from instead of manual for/yield loops. Clearer delegation and return propagation.
Bound infinite iterators with islice or break conditions. count() and cycle() never stop alone.

C - Decorators

Always apply @functools.wraps on wrapper functions. Preserves __name__, docs, and annotations.
Keep decorator stacks shallow and ordered consistently. Document classmethod/staticmethod ordering.
Parametrize via decorator factories (def retry(n):). Avoid global mutable config in decorators.
Do not swallow exceptions unless decorator purpose is suppression. Re-raise or chain with from.
Prefer FastAPI middleware/Depends for request scope. Function decorators for pure utilities.

D - functools and Caching

Cache only pure functions with hashable arguments (lru_cache). Call cache_clear when inputs change externally.
Set maxsize on lru_cache for unbounded key spaces. Prevent memory leaks.
Use singledispatch for open extension by type. Not long isinstance chains in one module.
partial for binding config, not for hiding required parameters. Keep public signatures readable.

E - Context Managers

Use with for files, locks, DB transactions, and temp config. Not bare open without close path.
Use ExitStack when number of contexts is runtime-determined. Dynamic file lists, plugin stacks.
Return False from __exit__ unless deliberately suppressing. Hidden failures corrupt state.
Prefer @contextmanager for simple setup/teardown under 15 lines. Class-based for reusable objects.
Test rollback paths by injecting exceptions inside with block.

FAQs

generator or list comprehension?

Generator when stream large or single-pass. List when need len, reuse, or indexing.

how many pipeline stages?

Split when each stage has name-worthy responsibility - avoid 10-line anonymous nest.

decorator in library public API?

Document stacking order and metadata preservation - consumers rely on name in logs.

lru_cache on methods?

Cache per-function; method args include self - consider function outside class or __hash__ on immutable self.

tee iterator?

Only when must fork single pass - memory grows with divergence.

async generator practices?

See asyncio section - use async with for acquisition, async for for consumption.

testing lazy pipelines?

list(pipeline(data)) in tests; assert intermediate cleanup with mocks on close.

reduce still ok?

Fine for rare folds; explicit loop often clearer for teams new to functional style.

class decorator vs inheritance?

Decorator for registration/metadata; inheritance for shared behavior implementation.

biggest iterator mistake?

Materializing giant intermediate lists between map/filter stages - stay lazy until boundary.

Generators & yield - lazy pipelines
Writing Decorators - wraps and factories
Context Managers - cleanup protocol
Comprehensions & Generator Expressions - genexp basics

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Iterators & Decorators Best Practices

Guidelines for lazy iteration, decorator hygiene, and composable pipelines - the patterns that keep Python services memory-safe and debuggable.

How to Use This List

Review when building ETL pipelines or log processors.
Apply in decorator-heavy frameworks (auth, retry, metrics).
Pair with profiling when iterator chains grow deep.
Use in code review for lambda in loops and missing @wraps.

A - Iteration

Use generators or generator expressions for streams larger than memory. Materialize only at final consumer when needed.
Compose pipelines as small generator functions. Each stage does one transform - easier testing.
Prefer itertools for chain, islice, groupby, product. Battle-tested and lazy.
Sort before groupby when grouping by key. Consecutive-key requirement is easy to forget.
Document single-pass iterators. Callers must not assume replay without tee or list().

B - Generators and Resources

Use try/finally or @contextmanager inside generators holding files/sockets. Consumers may stop early.
Close generators with .close() in tests when cleanup matters. Triggers GeneratorExit at yield.
Avoid side effects in generator definition - effects run at consumption. Lazy semantics surprise newcomers.
Use yield from instead of manual for/yield loops. Clearer delegation and return propagation.
Bound infinite iterators with islice or break conditions. count() and cycle() never stop alone.

C - Decorators

Always apply @functools.wraps on wrapper functions. Preserves __name__, docs, and annotations.
Keep decorator stacks shallow and ordered consistently. Document classmethod/staticmethod ordering.
Parametrize via decorator factories (def retry(n):). Avoid global mutable config in decorators.
Do not swallow exceptions unless decorator purpose is suppression. Re-raise or chain with from.
Prefer FastAPI middleware/Depends for request scope. Function decorators for pure utilities.

D - functools and Caching

Cache only pure functions with hashable arguments (lru_cache). Call cache_clear when inputs change externally.
Set maxsize on lru_cache for unbounded key spaces. Prevent memory leaks.
Use singledispatch for open extension by type. Not long isinstance chains in one module.
partial for binding config, not for hiding required parameters. Keep public signatures readable.

E - Context Managers

Use with for files, locks, DB transactions, and temp config. Not bare open without close path.
Use ExitStack when number of contexts is runtime-determined. Dynamic file lists, plugin stacks.
Return False from __exit__ unless deliberately suppressing. Hidden failures corrupt state.
Prefer @contextmanager for simple setup/teardown under 15 lines. Class-based for reusable objects.
Test rollback paths by injecting exceptions inside with block.

FAQs

generator or list comprehension?

Generator when stream large or single-pass. List when need len, reuse, or indexing.

how many pipeline stages?

Split when each stage has name-worthy responsibility - avoid 10-line anonymous nest.

decorator in library public API?

Document stacking order and metadata preservation - consumers rely on name in logs.

lru_cache on methods?

Cache per-function; method args include self - consider function outside class or __hash__ on immutable self.

tee iterator?

Only when must fork single pass - memory grows with divergence.

async generator practices?

See asyncio section - use async with for acquisition, async for for consumption.

testing lazy pipelines?

list(pipeline(data)) in tests; assert intermediate cleanup with mocks on close.

reduce still ok?

Fine for rare folds; explicit loop often clearer for teams new to functional style.

class decorator vs inheritance?

Decorator for registration/metadata; inheritance for shared behavior implementation.

biggest iterator mistake?

Materializing giant intermediate lists between map/filter stages - stay lazy until boundary.

Generators & yield - lazy pipelines
Writing Decorators - wraps and factories
Context Managers - cleanup protocol
Comprehensions & Generator Expressions - genexp basics

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Iterators & Decorators Best Practices

Search Documentation

Iterators & Decorators Best Practices