The Observability Correlation Model
Every page in this section can look like a shopping list of tools - structlog for logs, prometheus_client for metrics, OpenTelemetry for traces, Sentry for errors - each with its own library and its own setup steps.
Observability Basics already walks through that syntax across all four.
What ties them into one coherent practice instead of four unrelated integrations is a model: the same underlying events get observed from different angles, correlated by shared identifiers, and each angle has its own cost structure that decides how much of it you can afford to keep.
Understanding that model changes which pillar you reach for first when something breaks, and why "just log everything" or "just trace everything" both eventually stop working as a strategy.
Summary
- Logs, metrics, and traces are three different views of the same request or job, tied together by shared correlation identifiers, not three independent sources of truth.
- Insight: Without a shared identifier threading through all three, an incident investigation degenerates into guessing which log lines, which dashboard spike, and which slow trace all belong to the same event.
- Key Concepts: correlation ID, cardinality, signal vs. noise, liveness vs. readiness, the three questions (what happened, how often and how fast, where did the time go).
- When to Use This Model: Deciding which pillar to add instrumentation to, designing what a log line or span attribute should carry, and structuring an incident response runbook.
- Limitations/Trade-offs: Each pillar has a real cost - logs cost storage and query time, metrics cost cardinality, traces cost sampling decisions - so "instrument everything" is not a free strategy at any nontrivial scale.
- Related Topics: structured logging, the RED and USE methods, distributed tracing, health and readiness probes.
Foundations
A useful starting question is not "which observability tool should I use" but "what question am I actually trying to answer."
Logs answer "what specifically happened" - a discrete, timestamped event with arbitrary detail, like "order 4821 failed validation: missing shipping address."
Metrics answer "how often, and how fast, in aggregate" - a number tracked over time, like a request count or a p99 latency, without any single request's specific detail.
Traces answer "where did the time actually go, for this one request, across every service it touched" - the causal, hierarchical path of one operation.
A fourth, smaller pillar - health checks - answers a different kind of question entirely: not "what happened" but "is this instance able to serve traffic right now," a present-tense operational question the other three do not directly answer.
None of these four is a replacement for another; a metric can tell you error rate spiked at 14:02, but only a correlated log or trace can tell you why that specific spike happened.
Mechanics & Interactions
The mechanism that turns four separate signals into one coherent investigation is the correlation identifier - most commonly a request_id generated at the edge of a request and a trace_id propagated by a tracing system.
from contextvars import ContextVar
request_id: ContextVar[str] = ContextVar("request_id", default="-")
# every log line emitted during this request includes request_id,
# so a log search for one ID reconstructs the whole request's storyOnce that identifier is attached to every log line for a request, present as an attribute on the relevant trace spans, and available as a label (carefully, given cardinality limits below) for at least error-level metrics, an incident response follows a predictable path: a metric alert says something is wrong and roughly when, a log search scoped to the affected time window and request_id says specifically what happened, and a trace for that same ID says where the time or the failure actually occurred across services.
Cardinality is the single biggest cost lever across all three pillars, and it is easy to reason about wrongly.
A metric label with unbounded values - a raw user ID, a full URL with query parameters, a UUID - multiplies the number of distinct time series a metrics backend has to store, often turning a cheap counter into a storage and query problem; the same logic applies to span attributes in a tracing backend.
This is why Prometheus-style metrics use a small, bounded set of label values (method, status_code, endpoint) rather than per-request identifiers, and why high-cardinality detail belongs in logs and trace attributes instead, where it is expected and priced differently.
Sampling exists for the same underlying reason on the tracing side: recording every span for every request at real production traffic is expensive to store and query, so most tracing setups only keep a fraction of traces, usually biased toward keeping ones that errored or were slow.
Advanced Considerations & Applications
The RED method (Rate, Errors, Duration) and the USE method (Utilization, Saturation, Errors) are two common lenses for deciding what metrics to actually track, and picking the wrong one for a given component produces dashboards that look busy but do not answer operational questions.
RED fits request-driven services well, since rate, error rate, and duration per endpoint map directly onto what a caller experiences.
USE fits resources like connection pools, queues, and workers better, since utilization and saturation catch problems (a pool at 95% capacity) before they show up as errors at all.
Health checks interact with the other three pillars mostly at the boundary of an orchestrator, not inside an incident investigation: a readiness probe failing is itself often the first metric-like signal (probe failure rate) that triggers deeper log and trace investigation - see Health & Readiness for the liveness-versus-readiness distinction this depends on.
Error tracking tools like Sentry sit at an interesting intersection of logs and metrics: they aggregate exception events (log-like, full detail) into grouped issues with occurrence counts and trends (metric-like), which is why they tend to become the first place engineers look after a metric alert fires, ahead of raw log search.
As a system scales past a handful of services, the correlation model itself becomes the thing worth investing in deliberately - a consistent field name for the correlation ID across every service, log formatter, and tracing library, decided once, saves far more incident time than any single tool choice.
| Signal | Strength | Weakness | Best Fit |
|---|---|---|---|
| Logs | Arbitrary detail, cheap to add per event | Expensive to search at scale, unstructured logs resist correlation | Root-causing a specific known event |
| Metrics | Cheap to store and alert on in aggregate, great for dashboards | No per-request detail, cardinality limits what can be a label | Detecting that something is wrong and roughly how much |
| Traces | Shows exact causal path and timing across services | Sampling means some requests are never captured; needs propagation everywhere | Finding where time or failure occurred across a distributed call |
Common Misconceptions
- "More logging is always better observability." Past a point, high-volume unstructured logs become expensive to store and slow to search, actively hiding the signal that led you to add more logging in the first place.
- "Metrics and traces are redundant once you have one of them." A metric shows you a trend across all requests; a trace shows you one request's specific path - neither substitutes for the other, they answer different questions at different granularity.
- "A health check and a metrics dashboard tell you the same thing." A health check is a present-tense yes/no about one instance right now; a metrics dashboard shows aggregate trends over time across all instances, which is why an instance can be "unhealthy" for seconds before any dashboard trend reflects it.
- "Sampling traces means you'll miss important incidents." Well-designed sampling deliberately keeps a higher fraction of errored or slow traces than routine ones, so the traces most useful during an incident are exactly the ones least likely to be dropped.
- "Cardinality only matters for metrics." The same unbounded-label problem inflates cost in tracing backends too - a span attribute holding a raw user ID or full request body carries the same cardinality cost as a metric label would.
FAQs
What question does each observability pillar actually answer?
- Logs: what specifically happened, in arbitrary detail
- Metrics: how often and how fast, in aggregate, over time
- Traces: where did the time go, for one request, across every service
- Health checks: can this instance serve traffic right now
Why do correlation IDs matter so much for incident response?
Without a shared identifier connecting a log line, a trace, and a metric spike, there is no reliable way to confirm they describe the same underlying event rather than an unrelated coincidence in the same time window.
What is cardinality, in plain terms?
The number of distinct combinations of label or attribute values a metric or trace can have; a label with unbounded values (like a raw user ID) multiplies that number and drives up storage and query cost far faster than the same data added as a log field would.
Why can't I just put a request_id on every metric label?
A metric label is meant to have a small, bounded set of possible values so a backend can pre-aggregate efficiently; a request_id is effectively unbounded, which turns a cheap counter into a per-request time series and defeats the point of a metric.
How do RED and USE differ, and when do I use each?
RED (Rate, Errors, Duration) fits request-driven services where a caller's experience maps directly to those three numbers; USE (Utilization, Saturation, Errors) fits resources like pools, queues, and workers, catching saturation before it becomes visible errors.
Is sampling in tracing the same idea as sampling in logging?
They solve a similar cost problem but usually differently - tracing sampling decides which whole traces to keep (often biased toward errors and high latency), while log sampling, when used at all, usually just reduces volume of routine, low-value log lines.
Why does a health check need to be separate from a metrics dashboard?
A health check answers a present-tense question about one specific instance for an orchestrator's immediate routing decision, while a metrics dashboard aggregates trends across all instances over time - useful for humans investigating, not for a load balancer deciding whether to send the next request.
When should I add a new metric versus a new log field?
Add a metric when you need cheap, always-on aggregation and alerting over many events; add a log field when the value is high-cardinality or only useful when investigating a specific known event.
Why do error trackers like Sentry feel different from both logs and metrics?
They aggregate individual exception events (log-like detail) into grouped issues with occurrence trends (metric-like framing), which is why they often become the first stop after a metric alert, ahead of raw log search.
Does adopting distributed tracing make structured logging unnecessary?
No - traces show timing and causal structure well, but logs still carry the arbitrary business detail ("which field failed validation") that a trace's structured spans are not designed to hold.
What's the biggest mistake teams make when scaling observability?
Instrumenting more of everything without first agreeing on a consistent correlation identifier and field naming across every service, which makes it hard to actually connect the signals once there is enough of them to matter.
Is there such a thing as too much instrumentation?
Yes - past a point, excessive log volume, metric cardinality, or span creation adds real storage and query cost without a proportional increase in what you can actually learn from an incident.
Related
- Observability Basics - hands-on examples across logs, metrics, traces, and health
- Structured Logging - the log-side implementation of correlation identifiers
- Metrics - RED/USE metrics and the cardinality limits this page describes
- Distributed Tracing - trace propagation and sampling in depth
- Health & Readiness - the liveness-versus-readiness question this page separates from the other three pillars
- Observability Best Practices - the actionable-signal checklist distilled from this model
Stack versions: This page is conceptual and not tied to a specific stack version.