Async & Concurrency for Throughput
Python concurrency models target different bottlenecks: asyncio for I/O parallelism, multiprocessing for CPU parallelism, and threading for I/O with blocking libraries.
Recipe
import asyncio
import httpx
async def fetch_all(urls: list[str]) -> list[int]:
async with httpx.AsyncClient() as client:
responses = await asyncio.gather(*[client.get(u) for u in urls])
return [r.status_code for r in responses]When to reach for this:
- Many concurrent HTTP/database requests (I/O-bound)
- CPU-bound work on multiple cores (multiprocessing)
- Throughput matters more than single-request latency
Working Example
import asyncio
import httpx
from concurrent.futures import ProcessPoolExecutor
# I/O-bound: asyncio
async def fetch_urls(urls: list[str]) -> list[str]:
async with httpx.AsyncClient(timeout=10.0) as client:
tasks = [client.get(url) for url in urls]
responses = await asyncio.gather(*tasks, return_exceptions=True)
return [r.text[:100] if isinstance(r, httpx.Response) else str(r) for r in responses]
# CPU-bound: process pool
def compute(n: int) -> int:
return sum(i * i for i in range(n))
async def run_cpu_bound(items: list[int]) -> list[int]:
loop = asyncio.get_running_loop()
with ProcessPoolExecutor() as pool:
return list(await asyncio.gather(*[
loop.run_in_executor(pool, compute, n) for n in items
]))
async def main():
urls = ["https://httpbin.org/get"] * 20
io_results = await fetch_urls(urls)
cpu_results = await run_cpu_bound([100_000] * 4)
print(len(io_results), sum(cpu_results))
asyncio.run(main())What this demonstrates:
asyncio.gatherruns I/O concurrently on one threadProcessPoolExecutorbypasses GIL for CPU workreturn_exceptions=Trueprevents one failure from canceling all- Combine models: async orchestration + process pool for CPU stages
Deep Dive
Model Selection
| Bottleneck | Model | Tool |
|---|---|---|
| I/O (network, disk) | asyncio | httpx, asyncpg |
| CPU | multiprocessing | ProcessPoolExecutor |
| Blocking I/O library | threading | ThreadPoolExecutor |
| Mixed | asyncio + executor | run_in_executor |
Gotchas
- asyncio for CPU work - no speedup, adds complexity. Fix: multiprocessing or vectorization.
- Unbounded concurrency - overwhelms server or hits connection limits. Fix:
asyncio.Semaphore(50)to cap parallel requests. - Shared mutable state across threads - data races. Fix: queues, locks, or process isolation.
- Creating processes per task - fork overhead. Fix: reuse ProcessPoolExecutor.
- Blocking call in async function - freezes event loop. Fix:
await asyncio.to_thread(blocking_fn)or async library.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Sync + threading | Legacy blocking code | Greenfield async |
| Celery/RQ | Distributed task queue | In-process parallelism enough |
| Horizontal scaling | Production throughput | Local optimization phase |
FAQs
asyncio or threads?
asyncio for native async libraries. Threads for blocking libraries you cannot replace.
How many concurrent requests?
Start 10-50. Tune with semaphore. Watch server and client limits.
Does asyncio bypass the GIL?
No. asyncio multiplexes I/O on one thread. CPU parallelism needs multiprocessing.
FastAPI and asyncio?
FastAPI is async-native. Use async database drivers for full benefit.
How do I limit rate?
asyncio.Semaphore or token bucket with asyncio.sleep.
Process pool size?
Typically os.cpu_count(). I/O pools can be larger than CPU count.
asyncio.gather vs TaskGroup?
Python 3.14: prefer asyncio.TaskGroup for structured concurrency and error handling.
How do I benchmark throughput?
Requests per second under load (locust, hey, k6). Not single-request timeit.
Thread safety with free-threaded 3.14?
Audit shared mutable state. Use locks or immutable data structures.
When is sync enough?
Low traffic, few concurrent I/O operations, or batch jobs without latency requirements.
Related
- 30 Async Rules - asyncio pitfalls
- Concurrency - GIL and models
- Caching & Memoization - reduce I/O
- Performance Best Practices - model selection
Stack versions: This page was written for Python 3.14.0, FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.