Observability Best Practices

Actionable signals, not noise, mean on-call wakes up for customer-impacting problems - not for log volume or unbounded metric cardinality.

How to Use This List

Implement before declaring production-ready.
Review alert routing quarterly - delete noisy rules.
Pair with runbooks linking dashboards to remediation steps.

A - Logging

Emit JSON logs in production. One object per line with timestamp, level, event, service, environment.
Bind request_id on every HTTP request. Return X-Request-ID to clients.
Include trace_id/span_id in logs when tracing enabled. Jump between logs and traces in one click.
Never log secrets, tokens, or full card numbers. Allowlist fields; scrub in Sentry before_send.
Log business context as fields, not interpolated f-strings. Enables log platform filtering.

B - Metrics

Expose RED metrics per route template. Rate, errors, duration histograms.
Keep label cardinality low. No user IDs or raw URLs as labels.
Choose histogram buckets around SLO thresholds. p95/p99 visible in Grafana.
Protect /metrics endpoint. Internal network or auth - not public internet.
Add business counters sparingly. Small enum labels only (country, plan).

C - Traces and Errors

Sample traces in production (e.g. 5-20%). Bias toward errors if tail sampling available.
Instrument outbound HTTP and boto3. Missing spans hide downstream latency.
Set Sentry release to git SHA. Bisect regressions to deploy.
Alert on new Sentry issues in prod, not event count duplicates. Reduce fatigue.
Propagate trace context through queues. SQS/Kafka headers carry traceparent.

D - Health and Deploys

Separate /health/live and /health/ready. Deps checked only on readiness.
Readiness failure returns 503. Load balancer removes pod without restart loop on DB blip.
Startup probe for slow Python imports. ML models lazy-load or startup probe tolerates delay.
Smoke test health after deploy in CI/CD. Block promotion when failing.
Dashboard per service: RPS, 5xx, p95 latency, saturation. On-call default landing page.

E - Process and SLOs

Define SLOs with error budgets. Example: 99.9% availability monthly.
Page on budget burn rate, not single blip. Multi-window alerts reduce false pages.
Run load tests before major releases. Validate p99 under expected peak.
Document observability ownership per service. Who gets paged, which dashboards, which runbooks.
Review observability cost monthly. Drop unused high-cardinality metrics and 100% trace sampling.

FAQs

What is minimum viable observability?

JSON logs, health endpoints, RED metrics, Sentry for exceptions, one dashboard, one on-call runbook.

Logs vs metrics for alerts?

Metrics for thresholds and SLOs; logs for investigation after alert fires - not vice versa.

How much tracing?

Enough to debug weekly incidents - adjust sample rate vs observability bill.

Do scripts need metrics?

Cron scripts: structured logs + exit code monitoring; metrics optional for duration histogram if critical.

PII in observability?

Hash or omit; legal review for fields stored in Sentry and log retention period.

Who creates dashboards?

Service owners with platform template - consistency beats bespoke chart art.

How do I avoid duplicate alerts?

One alert per symptom routed to one team; correlate Sentry + Prometheus with routing rules.

Dev observability?

Pretty console logs locally; JSON in staging that mirrors prod pipeline.

OpenTelemetry mandatory?

De facto standard for new instrumentation - vendor agents secondary.

Biggest observability mistake?

Logging everything at INFO with no structure - cannot query, cannot alert, high cost.

Observability Basics - three pillars intro
Structured Logging - structlog patterns
Metrics - Prometheus instrumentation
Health & Readiness - probe design
Deployment Best Practices - prod rollout signals

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Observability Best Practices

Actionable signals, not noise, mean on-call wakes up for customer-impacting problems - not for log volume or unbounded metric cardinality.

How to Use This List

Implement before declaring production-ready.
Review alert routing quarterly - delete noisy rules.
Pair with runbooks linking dashboards to remediation steps.

A - Logging

Emit JSON logs in production. One object per line with timestamp, level, event, service, environment.
Bind request_id on every HTTP request. Return X-Request-ID to clients.
Include trace_id/span_id in logs when tracing enabled. Jump between logs and traces in one click.
Never log secrets, tokens, or full card numbers. Allowlist fields; scrub in Sentry before_send.
Log business context as fields, not interpolated f-strings. Enables log platform filtering.

B - Metrics

Expose RED metrics per route template. Rate, errors, duration histograms.
Keep label cardinality low. No user IDs or raw URLs as labels.
Choose histogram buckets around SLO thresholds. p95/p99 visible in Grafana.
Protect /metrics endpoint. Internal network or auth - not public internet.
Add business counters sparingly. Small enum labels only (country, plan).

C - Traces and Errors

Sample traces in production (e.g. 5-20%). Bias toward errors if tail sampling available.
Instrument outbound HTTP and boto3. Missing spans hide downstream latency.
Set Sentry release to git SHA. Bisect regressions to deploy.
Alert on new Sentry issues in prod, not event count duplicates. Reduce fatigue.
Propagate trace context through queues. SQS/Kafka headers carry traceparent.

D - Health and Deploys

Separate /health/live and /health/ready. Deps checked only on readiness.
Readiness failure returns 503. Load balancer removes pod without restart loop on DB blip.
Startup probe for slow Python imports. ML models lazy-load or startup probe tolerates delay.
Smoke test health after deploy in CI/CD. Block promotion when failing.
Dashboard per service: RPS, 5xx, p95 latency, saturation. On-call default landing page.

E - Process and SLOs

Define SLOs with error budgets. Example: 99.9% availability monthly.
Page on budget burn rate, not single blip. Multi-window alerts reduce false pages.
Run load tests before major releases. Validate p99 under expected peak.
Document observability ownership per service. Who gets paged, which dashboards, which runbooks.
Review observability cost monthly. Drop unused high-cardinality metrics and 100% trace sampling.

FAQs

What is minimum viable observability?

JSON logs, health endpoints, RED metrics, Sentry for exceptions, one dashboard, one on-call runbook.

Logs vs metrics for alerts?

Metrics for thresholds and SLOs; logs for investigation after alert fires - not vice versa.

How much tracing?

Enough to debug weekly incidents - adjust sample rate vs observability bill.

Do scripts need metrics?

Cron scripts: structured logs + exit code monitoring; metrics optional for duration histogram if critical.

PII in observability?

Hash or omit; legal review for fields stored in Sentry and log retention period.

Who creates dashboards?

Service owners with platform template - consistency beats bespoke chart art.

How do I avoid duplicate alerts?

One alert per symptom routed to one team; correlate Sentry + Prometheus with routing rules.

Dev observability?

Pretty console logs locally; JSON in staging that mirrors prod pipeline.

OpenTelemetry mandatory?

De facto standard for new instrumentation - vendor agents secondary.

Biggest observability mistake?

Logging everything at INFO with no structure - cannot query, cannot alert, high cost.

Observability Basics - three pillars intro
Structured Logging - structlog patterns
Metrics - Prometheus instrumentation
Health & Readiness - probe design
Deployment Best Practices - prod rollout signals

Stack versions: This page was written for Python 3.14.0 (stable 3.14, maintenance 3.13), FastAPI 0.115+, Django 5.2, Flask 3.1, Pydantic 2, PyTorch 2.6+, pandas 2.2+, Polars 1.x, ruff 0.9+, and uv 0.6+.

Observability Best Practices

Search Documentation

Observability Best Practices