Observability Highlights Summary

Every highlight bullet from the 9 pages in this section, gathered on one page and grouped by the page it came from.

The Observability Correlation Model

• Logs, metrics, and traces answer three different questions about the same events, not three copies of one answer
• Correlation IDs (request_id, trace_id) are what let you jump between pillars during an incident
• Cardinality, not data volume alone, is what makes metrics and traces expensive to store and query
• A health check answers a present-tense question none of the other three pillars can answer on their own
• More instrumentation is not automatically better observability past the point signal turns into noise

Observability Basics

• Logs explain what happened; metrics show how often and how fast
• Traces connect requests across services
• Structured JSON logs parse better than plain text
• Every production service needs health endpoints

Structured Logging

• Structured logging with JSON enables searching by user_id or trace_id
• Use keyword arguments in logs, not f-strings, to enable field indexing
• Never log secrets or PII without allowlisting and hashing ids in non-prod
• Bind OpenTelemetry trace context to correlate logs with APM and metrics
• Use JSONRenderer in production, ConsoleRenderer in development via env
• Huge payload dumps cause cost and noise; log size or hash instead

Metrics

• Label routes with template /users/{id} not raw IDs to avoid cardinality explosion
• Histogram buckets chosen around SLO thresholds prevent blind spots
• Counter status label enables error rate calculation from 5xx counts
• Secure /metrics endpoint with network policy or auth to prevent info leak
• Gunicorn sync workers require PROMETHEUS_MULTIPROC_DIR for accurate counts
• Add business metrics alongside RED metrics to debug user impact

Distributed Tracing

• Traces reveal latency accumulation across HTTP, databases, and cloud APIs
• Use probabilistic sampling 10% in production to control cost and overhead
• Inject trace_id into logs for dual-pane debugging across systems
• W3C traceparent header propagates context over HTTP automatically
• Instrument boto3 and SQS explicitly to trace AWS service latency
• Thread pool execution loses parent span context without propagation

Error Tracking

• Set release=GIT_SHA to bisect which deploy broke errors
• Capture unhandled exceptions and actionable errors only, not logs
• Use separate DSN or environments for dev and production instances
• Use send_default_pii=False to avoid unnecessary PII in Sentry
• Re-raise exceptions after capture so users get proper error responses
• Scrub auth headers and cookies from breadcrumbs to prevent PII leaks

Health & Readiness

• Split liveness and readiness checks to drain traffic without pod restart
• Liveness returns 200 if event loop runs, avoiding expensive kubelet restart
• Readiness returns 503 if dependencies unavailable to stop load balancer routing
• DB check on liveness kills all pods during outage, use readiness checks only
• Slow readiness probe blocks deploy finish, fix with startup probe instead
• Readiness flapping causes thundering herd on DB recovery, use cached results

Performance Monitoring (APM)

• Set SLOs on p95/p99 latency percentiles, not average, users notice the tails
• Align histogram buckets to SLO thresholds (300ms, 800ms) for visibility
• Use py-spy for production profiling; profile staging under load first
• Monitor Latency, Traffic, Errors, and Saturation, the four Golden Signals
• Sample traces strategically to reduce cost while tracking all request metrics
• Use error budgets to balance feature velocity with monthly SLO reviews

Observability Best Practices

• Structured logs with request_id and trace_id
• Metrics use low-cardinality labels
• Liveness and readiness are separate endpoints
• Alert on SLO burn, not every log ERROR