Working with APIs & Webhooks
Automation scripts integrate with SaaS and internal APIs via HTTP clients. Use httpx or requests with timeouts, retries, pagination loops, and optional webhook receivers for event-driven triggers.
Recipe
import httpx
with httpx.Client(timeout=30.0) as client:
resp = client.get(
"https://api.example.com/v1/items",
headers={"Authorization": "Bearer TOKEN"},
params={"page": 1, "limit": 100},
)
resp.raise_for_status()
data = resp.json()When to reach for this:
- Polling when vendor lacks webhooks
- Webhook handler scripts behind reverse proxy or Lambda
- Bulk export APIs with cursor pagination
- Internal admin APIs for ops automation
Working Example
Paginated GET client with retry, plus minimal webhook signature verification stub.
import hashlib
import hmac
import time
from typing import Any, Iterator
import httpx
class ApiClient:
def __init__(self, base_url: str, token: str) -> None:
self._client = httpx.Client(
base_url=base_url,
headers={"Authorization": f"Bearer {token}"},
timeout=httpx.Timeout(30.0),
)
def list_items(self, page_size: int = 100) -> Iterator[dict[str, Any]]:
cursor: str | None = None
while True:
params = {"limit": page_size}
if cursor:
params["cursor"] = cursor
resp = self._client.get("/v1/items", params=params)
resp.raise_for_status()
payload = resp.json()
for item in payload.get("items", []):
yield item
cursor = payload.get("next_cursor")
if not cursor:
break
def close(self) -> None:
self._client.close()
def verify_webhook(body: bytes, signature: str, secret: str) -> bool:
expected = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature)
if __name__ == "__main__":
client = ApiClient("https://api.example.com", "dev-token")
for i, item in enumerate(client.list_items()):
if i >= 3:
break
print(item.get("id"))
client.close()What this demonstrates:
- Cursor pagination until
next_cursorabsent - Shared
httpx.Clientreuses connections across pages hmac.compare_digestprevents timing leaks on webhook verification
Deep Dive
Polling vs Webhooks
| Mode | Pros | Cons |
|---|---|---|
| Polling | Simple, no inbound port | Delay, rate limits |
| Webhooks | Real-time | Signature verify, retries, idempotency |
Resilience
- Timeouts on connect and read
- Retry idempotent GETs on 502/503 with backoff
- Respect
Retry-Afterheader on 429
Python Notes
import httpx
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=0.5))
def fetch(client: httpx.Client, url: str) -> httpx.Response:
resp = client.get(url)
resp.raise_for_status()
return respGotchas
- No timeout - hung scripts forever. Fix:
httpx.Timeoutalways set. - Trusting webhook body without verify - forged events. Fix: HMAC or mTLS per vendor docs.
- Not idempotent on webhook retries - duplicate rows. Fix: store processed event IDs.
- Loading all pages into RAM - OOM on large exports. Fix: generator pattern processing per page.
- Secrets in URL query params - logged by proxies. Fix: Authorization header only.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Official SDK | Vendor provides maintained Python SDK | Thin REST surface |
| SQS/Lambda webhook | Scale inbound events | Single cron poller enough |
| GraphQL client | API is GraphQL-only | Simple REST pagination |
FAQs
httpx vs requests?
httpx supports HTTP/2 and async; requests is sync-only but ubiquitous - both fine for scripts with timeouts.
How do I handle rate limits?
Read X-RateLimit-* headers, sleep or backoff on 429, reduce concurrency.
Offset vs cursor pagination?
Prefer cursor for large datasets - offset pagination breaks when data mutates during walk.
How do I run webhook receiver?
FastAPI/Flask endpoint or Lambda Function URL - validate signature before work.
How do I test HTTP scripts?
httpx.MockTransport or pytest-httpx fixtures - no network in unit tests.
Should scripts use async httpx?
Sync is simpler for cron; async when orchestrating many parallel downloads in one event loop.
How do I log requests?
Log method, URL path, status, duration - not Authorization header or full PII bodies.
What about pagination links?
Some APIs use RFC 5988 Link header - parse rel="next" instead of JSON cursor field.
How do I pass JSON POST?
client.post("/hooks", json={"ok": True}) sets content-type automatically.
How does this relate to scheduling?
Poll-based scripts are often cron-fired; webhooks reduce schedule frequency.
Related
- Scripting Basics - CLI and env config
- Scheduling & Cron - poll schedules
- Robust Scripts - retries and idempotency
- httpx & requests - HTTP client depth
- System Scripting Best Practices - API automation rules
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.