Observability Best Practices
Actionable signals, not noise, mean on-call wakes up for customer-impacting problems - not for log volume or unbounded metric cardinality.
How to Use This List
- Implement before declaring production-ready.
- Review alert routing quarterly - delete noisy rules.
- Pair with runbooks linking dashboards to remediation steps.
A - Logging
- Emit JSON logs in production. One object per line with timestamp, level, event, service, environment.
- Bind request_id on every HTTP request. Return
X-Request-IDto clients. - Include trace_id/span_id in logs when tracing enabled. Jump between logs and traces in one click.
- Never log secrets, tokens, or full card numbers. Allowlist fields; scrub in Sentry
before_send. - Log business context as fields, not interpolated f-strings. Enables log platform filtering.
B - Metrics
- Expose RED metrics per route template. Rate, errors, duration histograms.
- Keep label cardinality low. No user IDs or raw URLs as labels.
- Choose histogram buckets around SLO thresholds. p95/p99 visible in Grafana.
- Protect
/metricsendpoint. Internal network or auth - not public internet. - Add business counters sparingly. Small enum labels only (
country,plan).
C - Traces and Errors
- Sample traces in production (e.g. 5-20%). Bias toward errors if tail sampling available.
- Instrument outbound HTTP and boto3. Missing spans hide downstream latency.
- Set Sentry
releaseto git SHA. Bisect regressions to deploy. - Alert on new Sentry issues in prod, not event count duplicates. Reduce fatigue.
- Propagate trace context through queues. SQS/Kafka headers carry traceparent.
D - Health and Deploys
- Separate
/health/liveand/health/ready. Deps checked only on readiness. - Readiness failure returns 503. Load balancer removes pod without restart loop on DB blip.
- Startup probe for slow Python imports. ML models lazy-load or startup probe tolerates delay.
- Smoke test health after deploy in CI/CD. Block promotion when failing.
- Dashboard per service: RPS, 5xx, p95 latency, saturation. On-call default landing page.
E - Process and SLOs
- Define SLOs with error budgets. Example: 99.9% availability monthly.
- Page on budget burn rate, not single blip. Multi-window alerts reduce false pages.
- Run load tests before major releases. Validate p99 under expected peak.
- Document observability ownership per service. Who gets paged, which dashboards, which runbooks.
- Review observability cost monthly. Drop unused high-cardinality metrics and 100% trace sampling.
FAQs
What is minimum viable observability?
JSON logs, health endpoints, RED metrics, Sentry for exceptions, one dashboard, one on-call runbook.
Logs vs metrics for alerts?
Metrics for thresholds and SLOs; logs for investigation after alert fires - not vice versa.
How much tracing?
Enough to debug weekly incidents - adjust sample rate vs observability bill.
Do scripts need metrics?
Cron scripts: structured logs + exit code monitoring; metrics optional for duration histogram if critical.
PII in observability?
Hash or omit; legal review for fields stored in Sentry and log retention period.
Who creates dashboards?
Service owners with platform template - consistency beats bespoke chart art.
How do I avoid duplicate alerts?
One alert per symptom routed to one team; correlate Sentry + Prometheus with routing rules.
Dev observability?
Pretty console logs locally; JSON in staging that mirrors prod pipeline.
OpenTelemetry mandatory?
De facto standard for new instrumentation - vendor agents secondary.
Biggest observability mistake?
Logging everything at INFO with no structure - cannot query, cannot alert, high cost.
Related
- Observability Basics - three pillars intro
- Structured Logging - structlog patterns
- Metrics - Prometheus instrumentation
- Health & Readiness - probe design
- Deployment Best Practices - prod rollout signals
Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.