What Is Observability? A Practical Primer (Part 1 of 3)

This is the first post in a three-part series on observability. In this post we'll cover what observability actually means and the three types of telemetry that make it possible. Part 2 looks at the tools — OpenTelemetry, Prometheus, and Splunk — and Part 3 puts them together into a working architecture.

It's 2 AM. Your phone buzzes. Checkout is failing for some users, and the on-call dashboard shows a spike in errors. The question that decides whether you're back in bed in twenty minutes or debugging until sunrise is simple: can you figure out what's happening inside your system without deploying new code to find out?

That, in one sentence, is observability.

Observability vs. monitoring

The two terms get used interchangeably, but there's a useful distinction.

Monitoring answers questions you thought to ask in advance. You decided CPU usage matters, so you built a CPU graph. You decided error rate matters, so you set an alert at 1%. Monitoring is great at telling you that something is wrong.

Observability is about answering questions you didn't anticipate. Why are only users in one region affected? Why does this one endpoint slow down every day at 4 PM? Why did this specific request fail when the previous thousand succeeded? Observability is about telling you why something is wrong.

The term comes from control theory: a system is "observable" if you can infer its internal state purely from its external outputs. For software, those outputs are telemetry — and telemetry comes in three flavors, commonly called the three pillars.

Pillar 1: Metrics

Metrics are numeric measurements aggregated over time. Request rate. Error count. 95th-percentile latency. Memory usage. Queue depth.

A metric looks something like this:

http_requests_total{service="checkout", status="500"} = 1,204

A name, some labels that describe what is being measured, and a number. Sample it every 15 seconds and you get a time series you can graph, compare, and alert on.

Metrics are cheap to store and fast to query because they're aggregates — you're not keeping every event, just counts and distributions. That's their superpower and their limitation: metrics tell you "error rate on checkout jumped from 0.1% to 4% at 02:03", but they can't tell you anything about one specific failed request. The individual events have been rolled up and thrown away.

Metrics answer: "Is something wrong? How big is it? When did it start?"

Pillar 2: Logs

Logs are timestamped records of discrete events. Every developer knows them — they're the oldest form of telemetry there is.

2026-07-25T02:03:41Z ERROR checkout-svc payment authorization failed
  user_id=8841 order_id=A-2210 gateway=stripe error="card_declined: insufficient_funds"

Where metrics are aggregates, logs are individuals. Each line captures rich, arbitrary detail about one moment in one process. When you need to know exactly what happened — the error message, the stack trace, the input that broke things — logs are where you go.

The trade-off is volume. A busy system emits millions of log lines per hour, which makes logs expensive to store and slow to search unless you have serious tooling behind them (more on that in Part 2). Structured logging — emitting JSON with consistent fields instead of free-form text — makes them dramatically more useful.

Logs answer: "What exactly happened in this specific event?"

Pillar 3: Traces

Traces are the newest pillar, and the one that only becomes essential once you leave the monolith behind.

In a microservices architecture, a single user action — say checkout — might touch fifteen services: the API gateway, auth, cart, inventory, pricing, payment, notifications, and so on. If that request takes 8 seconds, which service ate the time? Metrics tell you every service looks roughly healthy. Logs give you fifteen separate log streams with no obvious connection between them.

A distributed trace stitches the journey back together. Every request gets a unique trace ID that's passed along through every hop. Each unit of work (an HTTP call, a database query) records a span — its start time, duration, and parent. Assemble the spans and you get a waterfall view of the entire request:

place-order  ──────────────────────────────────── 8.2s
 ├─ auth-check ──                                 0.1s
 ├─ inventory ────                                0.3s
 ├─ pricing ───                                   0.2s
 └─ payment ────────────────────────────────────  7.4s   ← there's your problem
     └─ gateway-call (3 retries) ──────────────   7.2s

Traces answer: "Where in the request path did things go wrong or slow down?"

The pillars work together

Here's the key insight most introductions miss: the pillars aren't competing options — they're different zoom levels on the same incident, and a good workflow moves between them:

1. A metric alert fires: checkout error rate is spiking. (You know something is wrong.)

2. You pull up traces for failed checkout requests and see they all stall in the payment service on calls to the card gateway. (You know where.)

3. You jump to the logs for one of those exact requests — linked by its trace ID — and find the gateway is returning TLS handshake errors since a certificate rotated. (You know why.)

Twenty minutes, back to bed.

That workflow — alert on metrics, localize with traces, explain with logs — is the whole reason observability tooling exists. The catch is that no single tool does all three well, which is why a real observability stack is always a combination of tools.

In Part 2, we'll meet the three names you'll hear most often — OpenTelemetry, Prometheus, and Splunk — and see exactly which job each one does and why.