Closures, Protocols & Lazy Evaluation
Iterators, generators, decorators, and context managers look like four different features, but they share two underlying ideas: protocols (objects implementing a small, agreed-upon set of dunder methods) and closures (functions that carry state from an enclosing scope between calls).
for delegates to the iterator protocol, with delegates to the context manager protocol, and @decorator is just a closure wrapping a function - none of these are special-cased by the language, they're all ordinary objects that happen to implement the right hooks.
This page is the model underneath Iterators & the Iterator Protocol, Generators & yield, Context Managers, and Writing Decorators; once it clicks, itertools and functools read as a toolbox built on these same two ideas rather than a grab-bag of unrelated utilities.
Summary
- Python's
for,with, and@decoratorsyntax are all sugar over protocols - objects implementing specific dunder methods - and closures are how those objects (or plain functions) carry state between calls. - Insight: Once protocols and closures are visible as the shared mechanism, generators, decorators, and context managers stop being separate topics to memorize and become three applications of one idea.
- Key Concepts: iterator protocol, lazy evaluation, closure / free variable, context manager protocol, higher-order function.
- When to Use: Building a custom iterable, writing a decorator that needs to remember state across calls, or deciding between a generator and a list when processing large or infinite data.
- Limitations/Trade-offs: Lazy, protocol-based objects (generators, iterators) trade random access and re-use for memory efficiency - most are single-pass and can't be indexed or restarted.
- Related Topics: the object model and dunder dispatch, function scope (LEGB), async iteration, memoization.
Foundations
An iterator is any object with a __next__ method that returns the next value or raises StopIteration when exhausted; an iterable is any object with an __iter__ method that returns such an iterator.
for item in obj: is entirely sugar over this: Python calls iter(obj) once to get an iterator, then calls next() on it repeatedly until StopIteration stops the loop - there is no other magic involved.
A generator function (any function containing yield) is the easiest way to get an iterator without writing a class: calling it doesn't run the function body at all, it immediately returns a generator object, and the body only executes up to the next yield each time next() is called on it.
That deferred execution is lazy evaluation: values are produced one at a time, on demand, instead of all at once up front - which is why a generator can represent an infinite sequence without ever running out of memory.
A closure is a nested function that references a variable from an enclosing function's scope; Python keeps that variable alive in a shared cell object for as long as the closure exists, which is how a decorator or a generator-like function can remember state between separate calls without a class.
Mechanics & Interactions
The mechanic that trips people up most is that closures capture a reference to a variable's cell, not a snapshot of its value at definition time - which is why a loop that creates several closures over a loop variable often surprises people by having them all see the loop's final value.
def make_multiplier(factor):
def multiply(x):
return x * factor # 'factor' is read from the enclosing cell, live
return multiply
double = make_multiplier(2)
triple = make_multiplier(3)
print(double(5), triple(5)) # 10 15 - each closure keeps its own 'factor' cellwith obj: is sugar over the context manager protocol: Python calls obj.__enter__() before the block and guarantees obj.__exit__(exc_type, exc, tb) runs after it, even if the block raised an exception - which is exactly why with is the idiomatic way to guarantee cleanup (closing a file, releasing a lock) instead of a manual try/finally.
@contextlib.contextmanager builds a context manager out of a generator function instead of a class: code before the single yield becomes __enter__, code after it becomes __exit__, and any exception from inside the with block is raised at that yield point inside the generator, letting you handle it with an ordinary try/except around the yield.
A decorator is nothing more than a higher-order function: @log_calls above a function definition is exactly equivalent to writing my_func = log_calls(my_func) immediately after defining it, and the wrapper function it returns is a closure that remembers the original function so it can call it later, typically after or around some extra behavior.
yield from delegates iteration to a sub-generator or any iterable, forwarding its values (and, less commonly, sent values and exceptions) as if the outer generator had produced them itself - this is the mechanism Yield From & Sub-Generators covers in depth, and it exists specifically so generators can be composed without manual for ... yield boilerplate.
Advanced Considerations & Applications
itertools and functools don't introduce new mechanisms - they're composable functions built entirely on the iterator protocol and closures already described here: itertools.chain returns an iterator that lazily walks several iterables in sequence, and functools.lru_cache wraps a function in a closure that remembers previous arguments and results.
The single biggest architectural trade-off in this space is lazy versus eager evaluation: a list comprehension computes every element immediately and holds them all in memory, while a generator expression or itertools pipeline computes each element only when asked, at the cost of being single-pass and non-indexable.
itertools.batched (added in Python 3.12, with a strict keyword argument added in 3.13) is a recent example of this same lazy-composition philosophy: it turns any iterable into an iterator of fixed-size tuples without ever materializing the whole input in memory.
Async code extends the exact same protocol idea into a different execution model: async for uses __aiter__/__anext__ instead of __iter__/__next__, and async with uses __aenter__/__aexit__ - the protocol pattern is identical, only the methods are coroutines that can await inside them.
| Approach | Strength | Weakness | Best fit |
|---|---|---|---|
| List comprehension | Simple, indexable, reusable multiple times | Materializes everything in memory immediately | Small-to-medium data you'll iterate more than once |
| Generator expression / function | Constant memory regardless of length; supports infinite sequences | Single-pass only; no indexing or len() | Large or unbounded data, streaming/ETL pipelines |
| Custom class-based iterator | Full control over state, supports extra methods beyond iteration | More boilerplate than a generator for the same behavior | Reusable iterables that need to be restarted or expose extra API |
itertools composition | Lazy, memory-efficient, avoids manual loop bookkeeping | Reads less linearly for developers unfamiliar with the module | Combinatorics, chaining, slicing iterators without a full pass |
Decorators compound the same way at scale: stacking multiple decorators (@a then @b above a function) applies them bottom-up, so func = a(b(func)), and losing track of that order - or forgetting functools.wraps to preserve the original function's __name__/__doc__ - are the two most common sources of confusing decorator bugs in real codebases.
Common Misconceptions
- "A generator is a lazily-computed list." It's closer to a single-use iterator: you can't index it, check its length, or restart it once exhausted, unlike a list that happens to be computed lazily.
- "
forhas special built-in support for lists, dicts, and files."foronly ever callsiter()thennext()repeatedly - every iterable, built-in or custom, goes through the identical protocol. - "Closures copy the enclosing variable's value when the inner function is defined." They capture a live reference to the variable's cell, which is why closures created in a loop often all end up seeing that variable's final value unless it's explicitly captured per-iteration.
- "A decorator changes the function's actual source code." It replaces the name with a new (usually wrapper) function object; the original function object still exists, just referenced from inside the wrapper's closure.
- "
withjust runs some code before and after a block, like two function calls." It specifically guarantees__exit__runs even when the block raises an exception, and__exit__can inspect or even suppress that exception - a plain "before and after" pair of calls can't do either.
FAQs
What is the iterator protocol, concretely?
Two dunder methods: __iter__, which returns an iterator, and __next__ on that iterator, which returns the next value or raises StopIteration when there's nothing left - for loops are built entirely on calling these two methods.
Does calling a generator function run its code immediately?
No - calling it returns a generator object without running any of the function body; the body only executes up to the next yield each time next() is called on that object.
What does "lazy evaluation" mean in this context?
Values are produced one at a time, only when requested, instead of all being computed and stored up front - this is what lets a generator represent even an infinite sequence without exhausting memory.
What exactly does a closure capture - the value or the variable?
It captures a reference to the enclosing variable's cell, not a snapshot of its value at that moment, which is why the variable's current value is what a closure sees each time it runs, not whatever it was when the closure was created.
Why do closures made inside a loop sometimes all behave the same, unexpectedly?
Because they all share the same enclosing variable's cell rather than each getting their own copy, so by the time any of them actually runs, they all read whatever value that variable ended up holding after the loop finished.
What does `with obj:` actually guarantee?
That obj.__exit__() runs after the block completes, even if the block raised an exception - obj.__enter__() runs first and its return value is what as name binds to, if used.
Is `@contextlib.contextmanager` a shortcut, or a different mechanism entirely?
It's a shortcut over the same protocol: code before the generator's single yield becomes __enter__ behavior, and code after it becomes __exit__ behavior, with exceptions from the with block surfacing at that yield point.
What is a decorator, mechanically, in one sentence?
@decorator above a function definition is exactly equivalent to writing func = decorator(func) right after defining it - nothing more mysterious than a higher-order function call.
Why does `functools.wraps` matter when writing a decorator?
Without it, the wrapper function replaces the original's __name__, __doc__, and other metadata with its own generic values, which breaks introspection, debugging output, and documentation tools that expect to see the original function's identity.
Are `itertools` and `functools` a separate system from iterators and closures?
No - itertools functions return lazy iterators built on the same iterator protocol, and functools tools like lru_cache or partial are closures wrapping callables, so both modules are compositions of the two mechanisms this page describes.
When should I use a generator instead of a list comprehension?
When the data is large, unbounded, or only needs a single pass - a generator holds only its current state in memory, while a list comprehension materializes every element immediately, which costs more memory but allows re-use and indexing.
How does `async for` relate to the regular iterator protocol?
It's the same protocol pattern applied to coroutines: __aiter__/__anext__ replace __iter__/__next__, and both methods can await internally, letting iteration pause on actual I/O instead of only on synchronous computation.
Related
- Iterators & the Iterator Protocol -
__iter__/__next__and custom iterables in depth - Generators & yield - generator state and lazy pipelines up close
- Context Managers - the
__enter__/__exit__protocol andcontextlib - Writing Decorators - closures wrapping callables, argument handling, and stacking
- Closures & Scope Capture - the cell-capture mechanic in full detail
- itertools - lazy composition built on the iterator protocol
Stack versions: This page was written for Python 3.14 (stable) and Python 3.13 (maintenance), including
itertools.batched(added 3.12,strict=added 3.13).