Incident Response Highlights Summary
Every highlight bullet from the 8 pages in this section, gathered on one page and grouped by the page it came from.
- Incident response optimizes for restoring service under uncertainty - debugging optimizes for understanding, and the two goals can conflict
- Python incidents resolve fastest when triage moves top-down: infra, process (Gunicorn/Celery/event loop), application, business
- Mitigation and root cause are separate goals - a good response gets the first without waiting for the second
- The post-mortem is not paperwork - it's the mechanism that turns one incident into a permanently smaller blast radius
- Correlate logs with trace_id before changing code
- py-spy samples live stacks without restarting workers
- Reproduce prod config in a sandbox before hotfixes
- Capture request context: tenant, feature flags, deploy SHA
- Stabilize first; root cause second
- Declare severity in the first five minutes
- Assign IC, Tech Lead, Comms, and Scribe roles
- Mitigate before root cause during customer impact
- Python runbooks link deploy SHA and worker queues
- Post-incident review within five business days
- Focus on systems and process, not individuals
- Five Whys stops at controllable organizational fixes
- Every action item has an owner and due date
- Link deploy SHA, migrations, and queue depth in timeline
- Publish summaries customers can understand when required
- Memory leaks often hide in module-level caches
- Async services fail when sync I/O blocks the loop
- Dependency pins diverge between API and workers
- Cascading timeouts need deadline budgets
- Connection pool exhaustion looks like app bugs
- Rollback image SHA, not latest tag
- Pause workers before API rollback on schema changes
- Forward-fix when migration is irreversible
- Document data recovery RPO/RTO per service
- Practice rollback quarterly in game days
- Every post-mortem action item becomes measurable
- Regression tests reproduce the incident scenario
- Alerts fire on leading indicators, not only 5xx
- CI gates encode migration and dependency rules
- Review open guardrails monthly in ops review
- Rotations need backup and timezone fairness
- Every alert links to a runbook or gets deleted
- Page on symptoms customers feel, not every blip
- Track pages per engineer per week
- Compensate on-call with time off or pay
Reviewed by Chris St. John·Last updated Jul 31, 2026