Distributed Tracing on Kubernetes with Tempo and Grafana

Having traces is not the same as being able to use them. Plenty of teams instrument their services, point the exporter at something, and stop there — until an incident needs a specific slow request and the only tool available is grep-ing logs by timestamp and hoping. The gap between "traces exist" and "traces are useful during an incident" is entirely in the backend and the query surface, not the instrumentation.

This tutorial builds that backend. It deploys Grafana Tempo as a trace store, connects an OpenTelemetry Collector to it, and — the part most walkthroughs skip — configures Grafana so a span in a trace links directly to the matching log lines and the service's RED metrics. It also covers TraceQL, the query language that makes Tempo more than a place traces go to be forgotten, and tail sampling, which decides which traces are worth storing in the first place. If you haven't instrumented an application with the OTel SDK yet, do that first with OpenTelemetry Instrumentation Guide or Distributed Request Tracing with OpenTelemetry — this tutorial assumes spans are already being generated and picks up from the Collector onward.

What You'll Build

• Grafana Tempo deployed via the monolithic tempo Helm chart, with an OTLP receiver enabled
• An OpenTelemetry Collector otlp exporter pointed at Tempo, with a traces pipeline wired end to end
• A confirmed flow of real spans from a source app into Tempo
• A Tempo data source in Grafana with tracesToLogsV2 and tracesToMetrics correlation configured
• A working TraceQL query in Grafana Explore, filtering by service name and span duration
• A tailsamplingprocessor config in the Collector that keeps all errors and slow traces while sampling the rest

Step 1: Deploy Tempo

Tempo ships two chart shapes: tempo (monolithic, single binary) and tempo-distributed (separate distributor/ingester/querier/compactor deployments, backed by object storage). Use the monolithic chart here. The distributed chart is the right call once you're running it in production at real trace volume, but every extra component is another thing to configure and debug, and none of it changes how you query Tempo — TraceQL and the Grafana correlation setup are identical either way. Learn the query side on the simple deployment; move to tempo-distributed when scale, not learning, is the constraint.

Both charts moved out of the old grafana/helm-charts repo into a separate community-maintained repo in 2026. The old repo still has a tempo chart, which is the trap — helm search repo tempo against grafana/helm-charts returns a frozen copy (chart 1.24.4, app 2.9.0) rather than nothing at all, so an install from there succeeds and quietly gives you a Tempo that's several releases behind. Add the community repo and install from it explicitly:

1helm repo add grafana-community https://grafana-community.github.io/helm-charts
2helm repo update
3
4helm install tempo grafana-community/tempo \
5  --namespace observability --create-namespace \
6  --set tempo.receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317 \
7  --set tempo.receivers.otlp.protocols.http.endpoint=0.0.0.0:4318

Confirm it's up and note the service DNS name — you'll need it in the next step:

kubectl -n observability rollout status statefulset/tempo
kubectl -n observability get svc tempo

Local disk storage is the chart's default backend, which is fine for this tutorial. Traces don't survive a pod restart with local storage — production deployments back Tempo with S3, GCS, or Azure Blob via storage.trace.backend in the chart's values.

Step 2: Point the OpenTelemetry Collector at Tempo

If you already have a Collector running, add a traces pipeline exporting to Tempo's OTLP receiver. If you don't have one yet, OpenTelemetry Collector on Kubernetes covers deploying one from scratch — the config below is the piece that changes.

1# otel-collector-config.yaml
2receivers:
3  otlp:
4    protocols:
5      grpc:
6        endpoint: 0.0.0.0:4317
7      http:
8        endpoint: 0.0.0.0:4318
9
10processors:
11  batch: {}
12
13exporters:
14  otlp:
15    endpoint: tempo.observability.svc.cluster.local:4317
16    tls:
17      insecure: true
18
19service:
20  pipelines:
21    traces:
22      receivers: [otlp]
23      processors: [batch]
24      exporters: [otlp]

tls.insecure: true is fine inside the cluster network for this tutorial; a production Collector-to-Tempo hop should run over mTLS or at minimum a service mesh's mutual TLS, not plaintext gRPC.

Apply it and restart the Collector to pick up the new config:

kubectl -n observability create configmap otel-collector-config \
  --from-file=otel-collector-config.yaml -o yaml --dry-run=client | kubectl apply -f -
kubectl -n observability rollout restart deployment/otel-collector

Step 3: Generate Real Traces

You need something producing spans. Two options, pick whichever is closer to what you already have running:

Option A — an app you've already instrumented. Point its OTEL_EXPORTER_OTLP_ENDPOINT at the Collector:

kubectl set env deployment/your-app \
  OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc.cluster.local:4318

If you don't have an instrumented app yet, a Python service instrumented in a couple of lines works too:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

OTEL_SERVICE_NAME=checkout \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc.cluster.local:4318 \
opentelemetry-instrument python app.py

Option B — a trace-producing workload you can stand up in minutes. If you'd rather not build a demo app, Trace MCP Tool Calls With OpenTelemetry walks through instrumenting an MCP server — its traffic is a convenient source of realistic multi-hop spans (client call → tool call → downstream API) to send through the pipeline you just built.

Confirm spans are arriving before you touch Grafana. Check the Collector's own logs for accepted/exported span counts:

kubectl -n observability logs deployment/otel-collector | grep -i "traces"

And check Tempo's metrics endpoint for received spans, which confirms the Collector→Tempo hop specifically (as opposed to app→Collector):

kubectl -n observability port-forward svc/tempo 3200:3200
curl -s http://localhost:3200/metrics | grep tempo_distributor_spans_received_total

A non-zero, increasing counter here means the pipeline is intact end to end. If it's stuck at zero, the break is almost always the exporter endpoint in Step 2 — check it resolves and the port matches Tempo's receiver.

Step 4: Add Tempo as a Grafana Data Source

In Grafana: Connections → Data sources → Add data source → Tempo. Set the URL to Tempo's query frontend (the monolithic chart serves queries on the same service, port 3200):

http://tempo.observability.svc.cluster.local:3200

Saving here gets you basic trace search. The part worth real attention is the correlation config below it — this is what turns Tempo from "a place to paste a trace ID" into a tool you jump into mid-investigation from a log line or a metrics panel, and back out again.

Trace-to-logs

If you have Loki running, wire tracesToLogsV2 so that clicking a span opens the log lines from that exact service, at that exact time window:

1# Tempo data source config (provisioning YAML, or the equivalent UI fields)
2jsonData:
3  tracesToLogsV2:
4    datasourceUid: "loki-uid"          # the UID of your Loki data source
5    spanStartTimeShift: "-5m"
6    spanEndTimeShift: "5m"
7    tags: [{ key: "service.name", value: "service_name" }]
8    filterByTraceID: true
9    filterBySpanID: false
10    customQuery: false

spanStartTimeShift / spanEndTimeShift widen the log query window around the span's own duration — a 40ms span with no shift would search a 40ms log window, which is too narrow to reliably catch a log line whose timestamp resolution or ingestion delay pushes it a few seconds either side. The tags mapping is what makes this specific rather than generic: it tells Grafana "take this span's service.name attribute and use it as the service_name label filter in Loki," which is exactly the join key that connects a trace to its logs when you haven't put the trace ID into the log line itself.

Trace-to-metrics

1jsonData:
2  tracesToMetrics:
3    datasourceUid: "prometheus-uid"    # the UID of your Prometheus data source
4    spanStartTimeShift: "-1h"
5    spanEndTimeShift: "1h"
6    tags: [{ key: "service.name", value: "service" }]
7    queries:
8      - name: "Request rate"
9        query: 'sum(rate(http_server_duration_count{$$__tags}[5m]))'
10      - name: "Error rate"
11        query: 'sum(rate(http_server_duration_count{$$__tags,http_status_code=~"5.."}[5m]))'
12      - name: "P99 latency"
13        query: 'histogram_quantile(0.99, sum(rate(http_server_duration_bucket{$$__tags}[5m])) by (le))'

$$__tags is substituted with the tags mapping above at query time, so the three RED (rate, errors, duration) queries automatically scope to the service the span belongs to. The time shift here is wider than the logs one — an hour on each side — because the useful comparison is "what did this service's error rate look like around this incident," not a window scoped to a single span's few-hundred-millisecond duration.

Save the data source. From now on, opening any trace in Tempo shows a Logs for this span and Metrics for this span link next to each span, no manual re-querying required.

Step 5: Query with TraceQL

Open Explore, select the Tempo data source, and switch to the TraceQL query editor:

{ .service.name = "checkout" && span:duration > 200ms }

Current TraceQL scopes intrinsics explicitly — span:duration, trace:duration, span:status — rather than the bare duration shorthand older examples use; the scoped form is what current Tempo documentation shows.

This is a materially different search than tag-based search. A tag search (Tempo's older search UI, or Jaeger-style search) asks "show me traces that have this tag somewhere" — it can't express a relationship between two conditions on the same span, and it can't compare a numeric field like duration with an operator. TraceQL treats each span as a structured record and lets you filter and combine conditions the way you'd write a WHERE clause:

{ .service.name = "checkout" && span:status = error }

{ .service.name = "checkout" && span:duration > 200ms } | select(.http.status_code, .http.route)

{ .service.name = "payment-api" } >> { .service.name = "checkout" && span:duration > 500ms }

That last one uses TraceQL's structural operators (>> for descendant) to find traces where a slow checkout span was caused somewhere downstream of a payment-api span — the kind of question tag search has no vocabulary for at all, because it has no concept of span relationships within a trace.

Step 6: Sampling — Head vs Tail

Not every trace is worth storing, and the decision of which to keep can happen in two places:

Head sampling, decided in the SDK before the trace is even fully generated (TraceIdRatioBasedSampler in the OTel SDK, or similar). It's cheap — no buffering, no extra hop — but it decides blind. A sampler set to keep 10% of traces keeps 10% of your errors and 10% of your slow requests too, which are exactly the traces you'll want during an incident.

Tail sampling, decided in the Collector after a trace is complete, using the tailsamplingprocessor (contrib distribution only — otelcol-contrib, not core otelcol):

1processors:
2  tail_sampling:
3    decision_wait: 10s
4    num_traces: 50000
5    policies:
6      - name: keep-errors
7        type: status_code
8        status_code: { status_codes: [ERROR] }
9      - name: keep-slow
10        type: latency
11        latency: { threshold_ms: 500 }
12      - name: sample-the-rest
13        type: probabilistic
14        probabilistic: { sampling_percentage: 5 }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp]

This is usually the right default: keep 100% of errors and anything over 500ms, and sample 5% of everything else for baseline visibility. The cost is that decision_wait buffers all spans for a trace in memory until the Collector has seen the whole thing (or the wait expires), which needs more Collector memory than head sampling and requires all spans for one trace to land on the same Collector instance — routing that correctly across multiple Collector replicas needs a load balancer exporter in front, which is out of scope here but worth knowing before you scale this past a single Collector pod.

Step 7: Verify — Find a Slow Trace and Read the Waterfall

Close the loop. In Grafana Explore, with the Tempo data source and TraceQL selected:

{ .service.name = "checkout" && (span:duration > 500ms || span:status = error) }

Run it, and open one of the returned traces. You should see:

• A span waterfall with the root span at the top and child spans indented beneath it, width proportional to duration
• The specific downstream service hop that accounts for most of the total duration — not just "checkout was slow," but which call inside it was slow
• If you configured Step 4, a Logs for this span link on the slow span that opens Loki already filtered to that service and time window
• A Metrics for this span link showing that service's request/error/latency graphs around the same window

If the waterfall shows only one span with no children even though the request clearly crossed multiple services, that's a context propagation gap upstream of Tempo, not a Tempo problem — see the "Why do my traces show as separate traces" section in Distributed Request Tracing with OpenTelemetry.

Where to Go Next

• OpenTelemetry Instrumentation Guide — if the app you pointed at Tempo isn't instrumented yet, or you want to add more manual spans around business logic
• OpenTelemetry Collector on Kubernetes — deployment patterns for the Collector this tutorial's config builds on, including HA and resource sizing
• Distributed Request Tracing with OpenTelemetry — the app-instrumentation and context-propagation side, useful if traces are arriving broken or incomplete
• Trace MCP Tool Calls With OpenTelemetry — a concrete trace-producing workload to point this Tempo setup at instead of building a demo app from scratch