A Simple Observability Architecture with OTel, Prometheus, and Splunk (Part 3 of 3)
This is the final post in a three-part series on observability. Part 1 covered the three pillars — metrics, logs, and traces. Part 2 introduced OpenTelemetry, Prometheus, and Splunk and the distinct role each plays. Now let's wire them together.
By now the division of labor should feel natural: OpenTelemetry generates and routes telemetry, Prometheus stores and alerts on metrics, and Splunk stores and searches logs and traces. The architecture that connects them is refreshingly simple — one pipeline, one fork.
The architecture
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Service A │ │ Service B │ │ Service C │ ← your backend services,
│ (OTel SDK)│ │ (OTel SDK)│ │ (OTel SDK)│ instrumented with OTel
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
└───────────────┼───────────────┘
│ OTLP (traces + metrics + logs)
▼
┌──────────────────┐
│ OTel Collector │ ← batch, filter, enrich,
│ (receive/process │ sample, then route
│ /export) │ by signal type
└────────┬─────────┘
│
┌─────────────┴─────────────┐
│ metrics │ logs & traces
▼ ▼
┌─────────────┐ ┌─────────────┐
│ Prometheus │ │ Splunk │
│ (TSDB + │ │ (log search,│
│ alerting) │ │ APM/traces)│
└──────┬──────┘ └─────────────┘
│ PromQL
▼
┌─────────────┐
│ Grafana │ ← dashboards & alert visualization
└─────────────┘Following the data
Step 1 — Services emit telemetry. Each backend service carries the OTel SDK, usually via auto-instrumentation. Every incoming request produces spans; counters and histograms tick; log lines get stamped with the current trace ID. The service itself has no idea Prometheus or Splunk exist — it just speaks OTLP to a local endpoint.
Step 2 — The Collector processes and routes. The OTel Collector receives everything and runs it through per-signal pipelines: a memory limiter (so the Collector never OOMs), a batcher (for efficient export), an attribute processor (adding deployment.environment: production to everything), and perhaps a trace sampler. Then it forks: metrics are exported to Prometheus; logs and traces are exported to Splunk.
The routing lives in a few lines of Collector config:
yaml
service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, resource, batch]
exporters: [prometheusremotewrite] # → Prometheus
logs:
receivers: [otlp]
processors: [memory_limiter, resource, batch]
exporters: [splunk_hec] # → Splunk (HTTP Event Collector)
traces:
receivers: [otlp]
processors: [memory_limiter, resource, batch]
exporters: [otlphttp/splunk] # → Splunk APMThis config is the fork in the diagram. Each pillar gets its own pipeline, and each pipeline ends at the backend best suited to that data shape.
Step 3 — Storage and alerting. Prometheus stores the metrics as time series and continuously evaluates alert rules; Grafana renders them into dashboards. Splunk indexes every log line and assembles spans into complete traces.
The payoff: a 2 AM incident, end to end
Here's the workflow from Part 1, now mapped onto real tools:
02:03 — Prometheus fires an alert. The rule
rate(http_requests_total{status="500", service="checkout"}[5m]) > 0.01breaches. Alertmanager pages you.02:05 — Grafana scopes the blast radius. The checkout dashboard shows errors isolated to one region and one endpoint:
POST /payments. Latency p95 has jumped from 200 ms to 8 s.02:08 — Splunk traces localize the fault. You filter traces for failed
POST /paymentsrequests. The waterfall shows every one stalling in the payment service on an outbound call to the card gateway — 3 retries, then failure.02:11 — Splunk logs explain it. You click through from a trace to its log lines (linked by trace ID) and find
TLS handshake failed: certificate expired. A gateway certificate rotated at 02:00 and the payment service's trust store wasn't updated.02:15 — Fix deployed, error rate back to baseline on the Grafana dashboard.
Notice how each tool answered exactly the question it was built for: Prometheus said something's wrong, Grafana said here's how big, traces said here's where, logs said here's why. No single tool could have carried that whole chain alone — and thanks to OTel, none of them needed application changes to participate.
Notes for a real deployment
A few things worth knowing before you take this to production:
Prometheus can also scrape directly. Infrastructure metrics (node exporter, kube-state-metrics) are usually pulled by Prometheus the classic way, while app telemetry flows through the Collector. The two patterns coexist happily.
Scale the Collector in two tiers. On Kubernetes, the standard pattern is a DaemonSet agent on every node (cheap local collection and metadata enrichment) forwarding to a horizontally-scaled gateway Deployment (sampling, filtering, and the only place backend credentials live).
Swap-friendly by design. Because everything upstream speaks OTLP, replacing Splunk with Grafana Loki + Tempo — or the reverse — is a Collector config change, not a re-instrumentation project.
Observe the observers. The Collector exposes its own metrics (dropped spans, export failures, queue depth). Scrape them with Prometheus. A telemetry pipeline that fails silently is the most dangerous outage of all.
Wrapping up the series
Across these three posts we've gone from what observability is (the ability to explain your system's behavior from its outputs), through the three pillars (metrics for detection, traces for localization, logs for explanation), to the tools (OTel to generate, Prometheus to measure, Splunk to investigate) and finally an architecture that ties them into one pipeline.
The deepest takeaway is that observability isn't a product you buy — it's a data pipeline you design. Choose a vendor-neutral collection layer, match each data shape to a backend built for it, and make sure the signals link together (trace IDs everywhere!). Do that, and 2 AM gets a lot shorter.