threading
The threading module runs OS threads in one process with shared memory. Use threads for I/O-bound overlap; guard shared mutable state with Lock, RLock, Event, and Condition.
Recipe
import threading
counter = 0
lock = threading.Lock()
def inc() -> None:
global counter
with lock:
counter += 1When to reach for this:
- Blocking HTTP, DB, filesystem I/O in parallel
- Background workers in a sync app
- Timers and periodic tasks (
threading.Timer) - Signaling between threads (
Event) - Bridging sync libraries inside async via executor
Working Example
import threading
import time
from queue import Queue
results: Queue[int] = Queue()
stop = threading.Event()
def worker(worker_id: int) -> None:
while not stop.is_set():
time.sleep(0.05)
results.put(worker_id)
threads = [threading.Thread(target=worker, args=(i,), daemon=True) for i in range(3)]
for t in threads:
t.start()
time.sleep(0.2)
stop.set()
for t in threads:
t.join(timeout=1)
print("collected", results.qsize())What this demonstrates:
Eventsignals graceful shutdown- Daemon threads exit when main exits - use non-daemon for flush work
Queuepasses results without manual lock on listjoin(timeout)avoids hanging shutdown
Deep Dive
Primitives
| Type | Use |
|---|---|
| Lock | Mutual exclusion |
| RLock | Reentrant lock same thread |
| Event | One-shot or persistent flag |
| Semaphore | Limit concurrency count |
Thread Safety
- Prefer immutable messages through queues
- Protect invariants spanning multiple operations with locks
threading.local()for per-thread state- Deadlock: always acquire locks in consistent order
Gotchas
- Unlocked shared mutations - races corrupt data. Fix:
with lock:or queue. - Deadlock with nested locks - A then B vs B then A. Fix: lock ordering doc.
- Daemon threads mid-write - truncated output on exit. Fix: join non-daemon workers.
- CPU-bound threads for speed - GIL limits parallel bytecode. Fix: multiprocessing.
- Too many threads - context switch thrash. Fix: pool size ~ concurrent I/O waits.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| asyncio | High fan-out I/O, async APIs | Blocking SDK only |
| multiprocessing | CPU parallel | Need shared memory |
| concurrent.futures | Pool abstraction | Custom thread lifecycle |
FAQs
How many threads?
Start near concurrent blocking operations (connections), not CPU core count.
Lock vs RLock?
RLock when same thread re-enters guarded code (recursive helpers).
Are list.append thread-safe?
Single append is atomic in CPython but compound read-modify-write is not.
threading vs _thread?
Use threading - higher level, supports locks and join.
How do threads interact with asyncio?
loop.run_in_executor runs blocking code in thread pool without blocking loop.
What is thread-local storage?
threading.local() attributes isolated per thread - handy for request context in sync WSGI.
Can I share SQLite across threads?
Use same connection per thread or check sqlite3 thread safety mode - often serial access.
How do I name threads?
Thread(name="worker-1") improves debugger and log readability.
Are ThreadPoolExecutor threads reused?
Yes - pool amortizes thread creation cost.
How do I test threading code?
Stress tests + threading.Barrier; avoid flaky timing assertions - use events.
Related
- The GIL & Free-Threaded Python - CPU limits
- Queues & Producer/Consumer - safe handoff
- concurrent.futures - pools
- Sync/Async Bridging - executors
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.