OpenTelemetry, Prometheus, and Splunk: Who Does What in the O11y Stack (Part 2 of 3)

This is Part 2 of a three-part series on observability. Part 1 covered what observability is and the three pillars — metrics, logs, and traces. This post looks at the three tools you'll hear about most, and why each one owns a different piece of the problem. Part 3 assembles them into a working architecture.

Walk into any observability discussion and three names come up within the first five minutes: OpenTelemetry, Prometheus, and Splunk. A common beginner mistake is treating them as competitors — three products doing the same thing. They're not. They occupy three different stages of the telemetry pipeline:

GENERATE & COLLECT  →  STORE & QUERY  →  SEARCH & ANALYZE
  (OpenTelemetry)       (Prometheus)        (Splunk)

Let's take them one at a time.

OpenTelemetry: the instrumentation standard

OpenTelemetry (OTel) is the odd one out because it's not a monitoring product at all. There's no OTel dashboard, no OTel query language, no OTel storage. It's an open-source standard and toolkit — a CNCF project, like Kubernetes — for generating and moving telemetry data.

It gives you three things:

SDKs for your code. Libraries for Java, Python, Go, Node.js, .NET, and most other languages that let your application emit traces, metrics, and logs in a standardized format. Even better, OTel ships auto-instrumentation for common frameworks — point it at a typical Spring Boot or Express app and you get spans for every HTTP call and database query without writing a line of instrumentation code.

The Collector. A standalone agent that receives telemetry, processes it (batching, filtering noise, adding attributes like environment: prod, sampling traces), and exports it to any backend. Think of it as a programmable router for telemetry.

OTLP. The OpenTelemetry Protocol — a standard wire format, so telemetry isn't tied to any vendor's proprietary agent or API.

The reason OTel won the industry is vendor neutrality. Instrumentation is the most expensive part of observability — it lives inside your codebase. Before OTel, switching from vendor A to vendor B meant re-instrumenting everything. With OTel, you instrument once, and the Collector decides where the data goes. Swap Splunk for something else next year? Change a few lines of Collector config. Your applications never know the difference.

OTel's job: create the data and get it wherever it needs to go.

Prometheus: the metrics engine

Prometheus is an open-source time-series database and metrics system, and the de facto standard for metrics in the Kubernetes world (it was the second project ever adopted by the CNCF, right after Kubernetes itself).

Its model is distinctive:

• Pull-based collection. Instead of apps pushing data, Prometheus scrapes HTTP endpoints — typically /metrics — on a schedule. Your services expose their current numbers; Prometheus comes and reads them every 15–30 seconds.

• Labeled time series. Everything is stored as metric_name{label=value, ...} over time, e.g. http_requests_total{service="checkout", status="500"}. Labels make slicing trivial.

• PromQL. A genuinely powerful query language. "95th-percentile checkout latency over the last 5 minutes, per region" is one line of PromQL.

• Alertmanager. Rules evaluate continuously and fire alerts — with grouping, deduplication, and routing to Slack or PagerDuty — the moment a condition is breached.

Prometheus is usually paired with Grafana for dashboards, since Prometheus's built-in UI is deliberately bare-bones.

Why does Prometheus only do metrics?

This is a question worth pausing on, because the answer explains a lot about observability tooling in general. It's not a missing feature — it's a deliberate architectural choice, and it comes down to the shape of the data:

1. Storage model. Prometheus's entire engine is built around compressing regular, numeric samples — the same metric names and labels arriving at predictable intervals. That regularity is why it can store months of metrics in a tiny footprint and answer aggregate queries in milliseconds. Logs are the opposite: irregular, high-cardinality, arbitrary text. Storing log lines in a time-series database would be like storing novels in a spreadsheet.

2. The cardinality wall. Every unique label combination in Prometheus creates a new time series. Labels like user_id or request_id — exactly the fields logs and traces revolve around — would explode into millions of series and take the server down. Prometheus is fast because it refuses to track individuals.

3. Unix philosophy. The Prometheus team explicitly scoped the project to do one thing extremely well. Logs and traces were left to purpose-built systems.

So Prometheus can tell you "checkout errors jumped to 4% at 02:03" instantly and cheaply — but it has, by design, thrown away the individual events. For "why did this request fail," you need a different kind of engine.

Prometheus's job: store metrics, answer aggregate questions fast, and fire alerts.

Splunk: logs, traces, and deep analysis

Splunk is a commercial data analytics and observability platform, and it comes at the problem from the opposite direction: it was built from day one to ingest massive volumes of unstructured event data and make it searchable.

• Log management at scale is its historical superpower. Splunk indexes everything you send it — application logs, infrastructure logs, audit trails — and lets you search across billions of events with SPL (Search Processing Language), its query DSL. This is exactly the individual-event, high-cardinality workload Prometheus refuses to touch.

• Splunk Observability Cloud extends the platform into a full o11y suite: distributed tracing and APM (built on its SignalFx acquisition), real-time metrics, and infrastructure monitoring. Notably, its tracing product ingests 100% of traces rather than sampling at the agent, which matters when the one request you care about is the one that got dropped.

• Correlation is the payoff: because OTel stamps trace IDs into log lines, Splunk can jump from a slow trace straight to the exact log entries produced by that request.

• Splunk is also a major contributor to OpenTelemetry — its standard ingestion agent these days is the OTel Collector.

The trade-offs: Splunk is enterprise-grade and priced accordingly (typically by data ingested), and SPL has a learning curve. Open-source-minded teams often substitute Grafana Loki (logs) and Tempo or Jaeger (traces) in the same architectural slot — the role stays identical.

So why does Splunk get logs and traces?

Because logs and traces are structurally the same kind of data: individual, high-cardinality events with rich attributes, where the questions are "find me this specific one" and "show me everything related to it." That's an indexing-and-search problem — Splunk's home turf. Metrics are aggregates over time — a compression-and-math problem — which is Prometheus's home turf. The split isn't arbitrary; it follows the data.

Splunk's job: store the individual events, and let you dig into any one of them.

The one-table summary

OpenTelemetryPrometheusSplunkWhat it isOpen standard + toolkitOpen-source metrics TSDBCommercial analytics platformPillarsGenerates all threeMetrics only (by design)Logs & traces (+ metrics in Obs. Cloud)Data shape— (it's the pipe)Regular numeric aggregatesIrregular high-cardinality eventsQuestion it answers"How do I collect & route telemetry?""Is something wrong, and how big?""What exactly happened, and why?"CostFree (your engineering time)Free (self-hosted)Licensed, by ingest volume

Three tools, three jobs, zero overlap. The natural next question is: how do you actually wire them together? That's Part 3, where we build a simple end-to-end architecture and walk a real incident through it.