multiprocessing
multiprocessing spawns child processes with separate Python interpreters - the portable way to parallelize CPU-bound Python on default GIL builds. Share data via queues, pipes, or multiprocessing.shared_memory.
Recipe
from multiprocessing import Pool
def work(n: int) -> int:
return n * n
if __name__ == "__main__":
with Pool() as pool:
print(pool.map(work, range(5)))When to reach for this:
- CPU-heavy pure Python loops
- Parallel data transforms across cores
- Isolating failure domains (one worker crash does not kill all)
- Bypassing GIL for compute pipelines
- Batch jobs with independent tasks
Working Example
from multiprocessing import Process, Queue
def producer(q: Queue, count: int) -> None:
for i in range(count):
q.put(i * i)
def consumer(q: Queue) -> None:
items = []
while True:
item = q.get()
if item is None:
break
items.append(item)
print("sum", sum(items))
if __name__ == "__main__":
q: Queue = Queue()
p = Process(target=producer, args=(q, 5))
c = Process(target=consumer, args=(q,))
c.start()
p.start()
p.join()
q.put(None)
c.join()What this demonstrates:
Queuepasses picklable objects between processes- Sentinel
Noneends consumer loop if __name__ == "__main__"required for spawn start method (macOS, Windows)- Processes do not share in-memory objects by default
Deep Dive
Start Methods
| Method | Behavior |
|---|---|
| spawn | Fresh interpreter (default macOS/Win) |
| fork | Copy parent (Unix, careful with threads) |
IPC Options
Queue,Pipefor messagesshared_memoryfor large arrays (3.8+)- Prefer passing immutable snapshots over shared mutation
Gotchas
- Missing main guard - recursive spawn on import. Fix:
if __name__ == "__main__". - Unpicklable closures - worker target must be top-level or picklable. Fix:
initializer+ module functions. - Huge object pickling - slow IPC. Fix: shared memory or file-backed chunks.
- fork + threads - deadlocks possible. Fix: prefer spawn on macOS; start processes before threads.
- Oversubscribing Pool - memory × workers exhausts RAM. Fix:
maxtasksperchildand worker cap.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| concurrent.futures.ProcessPoolExecutor | Simpler API | Need Process primitives |
| subprocess one-shot | Shell commands | Python function farm |
| C extension / NumPy | Releases GIL in threads | Pure Python hot loop |
FAQs
Pool size?
os.cpu_count() starting point; lower if each task allocates large memory.
Why is spawn default on macOS?
Safer with frameworks and Objective-C runtime than fork after threads started.
Can processes share a dict?
Use Manager().dict() - slower; prefer message passing.
How do I log from workers?
Configure logging in worker initializer; avoid inherited stale handlers.
Process vs subprocess module?
multiprocessing runs Python functions; subprocess runs executables.
maxtasksperchild?
Recycles workers after N tasks to curb memory leaks in long pools.
How do I cancel work?
Executor shutdown(cancel_futures=True) (3.9+) or sentinel through Queue.
Are lambdas picklable?
No on spawn - use def at module level.
Does asyncio replace multiprocessing?
No - asyncio concurrent I/O; processes parallel CPU.
How to benchmark?
Wall time with realistic payload size; include spawn overhead in short tasks.
Related
- concurrent.futures - ProcessPoolExecutor
- The GIL & Free-Threaded Python - why processes
- Queues & Producer/Consumer - patterns
- subprocess - external commands
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.