multiprocessing

Q: Pool size?

os.cpu_count() starting point; lower if each task allocates large memory.

Q: Can processes share a dict?

Use Manager().dict() - slower; prefer message passing.

Q: How do I log from workers?

Configure logging in worker initializer; avoid inherited stale handlers.

Q: Process vs subprocess module?

multiprocessing runs Python functions; subprocess runs executables.

Q: How do I cancel work?

Executor shutdown(cancel_futures=True) (3.9+) or sentinel through Queue.

Q: Are lambdas picklable?

No on spawn - use def at module level.

multiprocessing spawns child processes with separate Python interpreters - the portable way to parallelize CPU-bound Python on default GIL builds. Share data via queues, pipes, or multiprocessing.shared_memory.

Recipe

from multiprocessing import Pool
 
def work(n: int) -> int:
    return n * n
 
if __name__ == "__main__":
    with Pool() as pool:
        print(pool.map(work, range(5)))

When to reach for this:

CPU-heavy pure Python loops
Parallel data transforms across cores
Isolating failure domains (one worker crash does not kill all)
Bypassing GIL for compute pipelines
Batch jobs with independent tasks

Working Example

from multiprocessing import Process, Queue
 
def producer(q: Queue, count: int) -> None:
    for i in range(count):
        q.put(i * i)
 
def consumer(q: Queue) -> None:
    items = []
    while True:
        item = q.get()
        if item is None:
            break
        items.append(item)
    print("sum", sum(items))
 
if __name__ == "__main__":
    q: Queue = Queue()
    p = Process(target=producer, args=(q, 5))
    c = Process(target=consumer, args=(q,))
    c.start()
    p.start()
    p.join()
    q.put(None)
    c.join()

What this demonstrates:

Queue passes picklable objects between processes
Sentinel None ends consumer loop
if __name__ == "__main__" required for spawn start method (macOS, Windows)
Processes do not share in-memory objects by default

Deep Dive

Start Methods

Method	Behavior
spawn	Fresh interpreter (default macOS/Win)
fork	Copy parent (Unix, careful with threads)

IPC Options

Queue, Pipe for messages
shared_memory for large arrays (3.8+)
Prefer passing immutable snapshots over shared mutation

Gotchas

Missing main guard - recursive spawn on import. Fix: if __name__ == "__main__".
Unpicklable closures - worker target must be top-level or picklable. Fix: initializer + module functions.
Huge object pickling - slow IPC. Fix: shared memory or file-backed chunks.
fork + threads - deadlocks possible. Fix: prefer spawn on macOS; start processes before threads.
Oversubscribing Pool - memory × workers exhausts RAM. Fix: maxtasksperchild and worker cap.

Alternatives

Alternative	Use When	Don't Use When
concurrent.futures.ProcessPoolExecutor	Simpler API	Need Process primitives
subprocess one-shot	Shell commands	Python function farm
C extension / NumPy	Releases GIL in threads	Pure Python hot loop

FAQs

Pool size?

os.cpu_count() starting point; lower if each task allocates large memory.

Why is spawn default on macOS?

Safer with frameworks and Objective-C runtime than fork after threads started.

Can processes share a dict?

Use Manager().dict() - slower; prefer message passing.

How do I log from workers?

Configure logging in worker initializer; avoid inherited stale handlers.

Process vs subprocess module?

multiprocessing runs Python functions; subprocess runs executables.

maxtasksperchild?

Recycles workers after N tasks to curb memory leaks in long pools.

How do I cancel work?

Executor shutdown(cancel_futures=True) (3.9+) or sentinel through Queue.

Are lambdas picklable?

No on spawn - use def at module level.

Does asyncio replace multiprocessing?

No - asyncio concurrent I/O; processes parallel CPU.

How to benchmark?

Wall time with realistic payload size; include spawn overhead in short tasks.

concurrent.futures - ProcessPoolExecutor
The GIL & Free-Threaded Python - why processes
Queues & Producer/Consumer - patterns
subprocess - external commands

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Recipe

from multiprocessing import Pool
 
def work(n: int) -> int:
    return n * n
 
if __name__ == "__main__":
    with Pool() as pool:
        print(pool.map(work, range(5)))

When to reach for this:

CPU-heavy pure Python loops
Parallel data transforms across cores
Isolating failure domains (one worker crash does not kill all)
Bypassing GIL for compute pipelines
Batch jobs with independent tasks

Working Example

from multiprocessing import Process, Queue
 
def producer(q: Queue, count: int) -> None:
    for i in range(count):
        q.put(i * i)
 
def consumer(q: Queue) -> None:
    items = []
    while True:
        item = q.get()
        if item is None:
            break
        items.append(item)
    print("sum", sum(items))
 
if __name__ == "__main__":
    q: Queue = Queue()
    p = Process(target=producer, args=(q, 5))
    c = Process(target=consumer, args=(q,))
    c.start()
    p.start()
    p.join()
    q.put(None)
    c.join()

What this demonstrates:

Queue passes picklable objects between processes
Sentinel None ends consumer loop
if __name__ == "__main__" required for spawn start method (macOS, Windows)
Processes do not share in-memory objects by default

Deep Dive

Start Methods

Method	Behavior
spawn	Fresh interpreter (default macOS/Win)
fork	Copy parent (Unix, careful with threads)

IPC Options

Queue, Pipe for messages
shared_memory for large arrays (3.8+)
Prefer passing immutable snapshots over shared mutation

Gotchas

Missing main guard - recursive spawn on import. Fix: if __name__ == "__main__".
Unpicklable closures - worker target must be top-level or picklable. Fix: initializer + module functions.
Huge object pickling - slow IPC. Fix: shared memory or file-backed chunks.
fork + threads - deadlocks possible. Fix: prefer spawn on macOS; start processes before threads.
Oversubscribing Pool - memory × workers exhausts RAM. Fix: maxtasksperchild and worker cap.

Alternatives

Alternative	Use When	Don't Use When
concurrent.futures.ProcessPoolExecutor	Simpler API	Need Process primitives
subprocess one-shot	Shell commands	Python function farm
C extension / NumPy	Releases GIL in threads	Pure Python hot loop

FAQs

Pool size?

os.cpu_count() starting point; lower if each task allocates large memory.

Why is spawn default on macOS?

Safer with frameworks and Objective-C runtime than fork after threads started.

Can processes share a dict?

Use Manager().dict() - slower; prefer message passing.

How do I log from workers?

Configure logging in worker initializer; avoid inherited stale handlers.

Process vs subprocess module?

multiprocessing runs Python functions; subprocess runs executables.

maxtasksperchild?

Recycles workers after N tasks to curb memory leaks in long pools.

How do I cancel work?

Executor shutdown(cancel_futures=True) (3.9+) or sentinel through Queue.

Are lambdas picklable?

No on spawn - use def at module level.

Does asyncio replace multiprocessing?

No - asyncio concurrent I/O; processes parallel CPU.

How to benchmark?

Wall time with realistic payload size; include spawn overhead in short tasks.

concurrent.futures - ProcessPoolExecutor
The GIL & Free-Threaded Python - why processes
Queues & Producer/Consumer - patterns
subprocess - external commands

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

multiprocessing

Search Documentation

multiprocessing