On-Call Health

On-call only works long-term when rotations are fair, alerts are actionable, and recovery time is protected. Python fleets drown in noise when every Celery retry and Gunicorn worker restart pages someone.

Recipe

Quick-reference recipe card - copy-paste ready.

## Alert hygiene checklist (monthly)
- [ ] Each firing alert has runbook URL in annotation
- [ ] No alert fired >3x last month without ticket
- [ ] SLO-based burn alerts replace static thresholds where possible
- [ ] Business-hours vs wake-up severity separated
- [ ] Pages per on-call shift tracked in dashboard

# Prefer SLO burn over raw 5xx
- alert: CheckoutErrorBudgetBurn
  expr: slo_error_budget_burn_rate{service="orders-api"} > 1
  for: 10m
  labels:
    severity: page
  annotations:
    runbook: https://github.com/acme/runbooks/orders-api.md

When to reach for this:

• Engineers avoid on-call slots or burn out
• Same alert pages weekly with no action
• MTTR good but morale poor
• Scaling team without scaling pager load

Working Example

"""on_call_metrics.py - simple pager load tracker for retros."""
from __future__ import annotations
 
from dataclasses import dataclass
from datetime import date
 
 
@dataclass
class ShiftStats:
    engineer: str
    week: date
    pages: int
    actionable: int  # led to customer-impacting work
    false_positive: int
 
 
def health_score(stats: ShiftStats) -> str:
    if stats.pages == 0:
        return "quiet"
    action_rate = stats.actionable / stats.pages
    if stats.pages > 10 and action_rate < 0.3:
        return "noisy_rotation"
    if stats.pages > 15:
        return "overloaded"
    return "healthy"
 
 
def main() -> None:
    sample = ShiftStats("ana", date(2026, 7, 7), pages=12, actionable=4, false_positive=8)
    print(sample.engineer, health_score(sample))
 
 
if __name__ == "__main__":
    main()

What this demonstrates:

• Track actionable vs false-positive pages per shift
• Health score highlights noisy rotations needing alert tuning
• Data drives monthly ops review, not anecdotes
• Runbook links in alert annotations reduce mean time to mitigate

Deep Dive

How It Works

• Rotation design - Primary + secondary, max consecutive weeks, timezone swap fairness.
• Alert tiers - Page (wake), ticket (next day), log (dashboard only).
• SLO-based paging - Multi-window burn rates reduce false positives vs static thresholds.
• Follow-the-sun - Optional for global fleets; handoff doc required.
• Recovery - Comp time or pay; no hero culture for preventable noise.

Sustainable Rotation Template

Python Fleet Noise Sources

Gotchas

• Alerting on log ERROR count - Third-party blips page everyone. Fix: Page on SLO burn with for: duration.
• No secondary - Vacation becomes outage risk. Fix: Always schedule backup with calendar integration.
• Runbooks in wiki only - 3am link rot. Fix: Repo runbook path in every alert annotation.
• Ignoring false-positive rate - Engineers mute channel. Fix: Monthly delete or fix alerts with >50% false positive.
• Same person always on-call - Bus factor and burnout. Fix: Rotation tool enforces fairness; manager audits quarterly.

Alternatives

FAQs

Related

• Incident Response Playbooks - what to do when paged
• Turning Incidents into Guardrails - reduce repeat pages
• Production Debugging - first response toolkit
• Metrics - SLO instrumentation
• Team Onboarding - ramp new owners

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.