Trace MCP Tool Calls With OpenTelemetry: A Hands-On Setup

An agent trace that stops dead at a single span labelled execute_tool, with no children underneath, isn't a slow trace — it's a broken one. The context never crossed into the MCP server. HTTP header propagation can't fix this: stdio has no headers at all, and Streamable HTTP's headers describe the transport, not the individual tool call riding inside it.

This tutorial fixes that using MCP's own propagation channel — the _meta field — and gives you a local stack to actually see the result, rather than taking it on faith.

What You'll Build

• A local Jaeger instance via Docker Compose, receiving OTLP directly
• An MCP client that injects trace context into _meta before calling a tool
• An MCP server that extracts that context and parents its own span on it
• A trace in Jaeger showing the client's outer span and the server's inner span linked as parent/child

Step 1: Run a Local Observability Stack

1# docker-compose.yml
2services:
3  jaeger:
4    image: jaegertracing/all-in-one:1.60
5    ports:
6      - "16686:16686"   # UI
7      - "4317:4317"     # OTLP gRPC
8      - "4318:4318"     # OTLP HTTP

docker compose up -d

Open http://localhost:16686 — that's the Jaeger UI, empty for now.

No separate OpenTelemetry Collector here: Jaeger's all-in-one image has accepted OTLP natively on 4317/4318 since v1.35, so for a local two-process demo it would only be a hop that can break. Add a Collector when you need what it's actually for — fanning out to more than one backend, tail sampling, or scrubbing attributes before they leave the host — and point OTLPSpanExporter at the Collector instead of at Jaeger.

Step 2: Install the OTel SDK

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc

Create a small shared setup module both the client and server will import:

1# tracing.py
2from opentelemetry import trace
3from opentelemetry.sdk.resources import Resource
4from opentelemetry.sdk.trace import TracerProvider
5from opentelemetry.sdk.trace.export import BatchSpanProcessor
6from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
7
8
9def configure_tracing(service_name: str) -> None:
10    provider = TracerProvider(resource=Resource.create({"service.name": service_name}))
11    provider.add_span_processor(
12        BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True))
13    )
14    trace.set_tracer_provider(provider)

Call configure_tracing("mcp-client") and configure_tracing("mcp-server") from the two processes respectively — different service names, so Jaeger shows them as distinct services in one trace.

Step 3: Understand Why the Trace Breaks Without This

Distributed tracing works because each hop passes a traceparent to the next. HTTP frameworks do this automatically via headers. MCP breaks both halves of that assumption: stdio has no headers to carry anything, and Streamable HTTP's headers belong to the HTTP request, not the MCP message inside it — a retry can split one tool call across several HTTP requests, and one response stream can carry several messages.

The fix is a reserved _meta property bag on JSON-RPC messages. As of MCP's 2026-07-28 spec, traceparent, tracestate, and baggage are reserved inside _meta for W3C Trace Context (SEP-414). Because _meta lives in the message body, it works identically over stdio and Streamable HTTP. The one detail that matters: _meta sits in the request's params, alongside name and arguments — never inside arguments itself.

Check your SDK before writing this by hand — the MCP Python SDK's v2 line ships this automatically as built-in OpenTelemetry middleware, extracting and propagating _meta trace context without any of the code below. If you're still on the pre-v2 SDK (pip install "mcp<2"), or on an SDK in another language that hasn't caught up yet, the code below is exactly what that v2 machinery does under the hood.

Step 4: Instrument the Client — Inject

1# client_call.py
2from opentelemetry import trace
3from opentelemetry.propagate import inject
4
5from tracing import configure_tracing
6
7configure_tracing("mcp-client")
8tracer = trace.get_tracer("agent")
9
10
11async def call_tool(session, tool_name: str, arguments: dict):
12    with tracer.start_as_current_span(f"execute_tool {tool_name}") as span:
13        span.set_attribute("gen_ai.operation.name", "execute_tool")
14        span.set_attribute("gen_ai.tool.name", tool_name)
15        span.set_attribute("gen_ai.tool.type", "function")
16
17        carrier: dict[str, str] = {}
18        inject(carrier)  # writes traceparent (+ tracestate/baggage if set)
19
20        return await session.call_tool(
21            tool_name,
22            arguments=arguments,
23            meta=carrier,  # lands in params._meta, beside arguments
24        )

Step 5: Instrument the Server — Extract

1# server_handler.py
2from opentelemetry import trace
3from opentelemetry.propagate import extract
4
5from tracing import configure_tracing
6
7configure_tracing("mcp-server")
8tracer = trace.get_tracer("mcp-server")
9
10
11async def handle_tool_call(name: str, arguments: dict, meta: dict | None):
12    parent = extract(meta or {})  # empty Context if nothing was sent
13
14    with tracer.start_as_current_span(f"execute_tool {name}", context=parent) as span:
15        span.set_attribute("gen_ai.tool.name", name)
16        result = await run_tool(name, arguments)
17        span.set_attribute("gen_ai.tool.call.id", result.call_id)
18        return result

Wire this into the server from Build a Kubernetes MCP Server in Python by wrapping each @mcp.tool() function's invocation with handle_tool_call, or check whether your SDK version already extracts _meta for you before you duplicate the work.

One failure mode to know about now, before you hit it later: if extract receives an empty carrier, it returns an empty context, and passing that as context= makes the server span a new root — you get two disconnected traces, not an error. That looks like "tracing is broken" but actually means "propagation isn't reaching the server." If you'd rather an un-propagated call nest under whatever ambient span exists instead of starting a second root, guard the extracted context and replace the first two lines of handle_tool_call with this:

1async def handle_tool_call(name: str, arguments: dict, meta: dict | None):
2    parent = extract(meta or {})
3
4    # An empty carrier yields a context whose span is invalid. Passing None
5    # instead makes the SDK fall back to the ambient context, so the span
6    # nests locally rather than becoming a detached root.
7    if not trace.get_current_span(parent).get_span_context().is_valid:
8        parent = None
9
10    with tracer.start_as_current_span(f"execute_tool {name}", context=parent) as span:
11        ...

Which behaviour you want is a real choice, not a detail: a detached root makes missing propagation obvious in Jaeger, while the fallback keeps traces tidy and lets the bug hide. Prefer the detached root while you're still getting propagation working.

Step 6: Run Both Sides and Generate a Trace

Start your MCP server, then run a small script that drives the client:

# run_demo.py
import asyncio
from client_call import call_tool
# ... connect `session` to your running MCP server per its transport ...

asyncio.run(call_tool(session, "get_pods", {"namespace": "kube-system"}))

python run_demo.py

Step 7: Verify the Trace Links Up

Open Jaeger (http://localhost:16686), select the mcp-client service, and find your trace. You're looking for exactly this shape — not two separate traces:

execute_tool get_pods        (service: mcp-client)  ← outer span
└── execute_tool get_pods    (service: mcp-server)   ← inner span, child of the above

If instead you see two root spans with no parent/child relationship between them, propagation isn't reaching the server — go back to Step 5 and confirm meta is actually arriving at handle_tool_call (print it before calling extract), and that you're passing context=parent into start_as_current_span, not dropping it.

The nested pair, once it's correct, tells you something real: the outer span is what the client observed, including serialization and queueing; the inner is what the server actually spent executing the tool. The gap between them is transport overhead — usually small, and worth investigating specifically when it isn't.

Step 8: Add the GenAI Attributes That Make This Useful

A linked trace with no attributes tells you that something happened, not what it cost. Add the standard GenAI semantic convention attributes so the trace answers real questions:

Once these are on the spans, "where did the 47 seconds go" decomposes into model latency (sum of chat spans), tool latency (inner execute_tool spans), and transport latency (outer minus inner) — without guessing from interleaved logs.

Pitfalls to Avoid

• Don't set gen_ai.tool.call.arguments or .result by default. A tool reading pod logs puts that content straight into your tracing backend, retained and readable by anyone with dashboard access — the same egress concern as giving an agent read-only Kubernetes access, arriving through a different door.
• Set span status on tool errors explicitly. A tool that fails but returns an error string to the model is not a failed span by default — call span.set_status(Status(StatusCode.ERROR)) yourself or your error rate will look better than it is.
• Sample whole traces, not individual spans. Span-level head sampling shreds agent traces into orphaned fragments. Sample at the root, or use Collector tail sampling to keep slow/errored traces and drop the boring ones.

Where to Go Next

You've verified the propagation mechanics on a minimal example — the next step is pointing this at a real MCP server and a real agent loop:

• Apply this to the server you built — Build a Kubernetes MCP Server in Python.
• Read the full span-shape and attribute reference — Trace MCP Tool Calls With OpenTelemetry covers the complete GenAI attribute table and the multi-turn trace shape for a full agent loop.
• If you haven't instrumented the model calls yet, do those first — they're usually the largest latency contributor and the fastest to add. The OpenTelemetry instrumentation guide covers the fundamentals this tutorial assumed.