Trace MCP Tool Calls With OpenTelemetry

Trace MCP Tool Calls With OpenTelemetry

An agent takes 47 seconds to answer. Somewhere in there it made four MCP tool calls, two model round-trips, and hit a database. Which one cost you the 47 seconds?

Without tracing you're reading interleaved logs from three processes and guessing. With naive tracing you get something worse: a confident-looking trace showing one 47-second span labelled execute_tool and no children, because the trace context never crossed into the MCP server.

That break is the subject of this post. It has a specific cause and a specific fix, and neither is "add the OTel SDK to both sides."

This assumes you know OpenTelemetry basics — if not, the instrumentation guide and running the Collector on Kubernetes cover the fundamentals. What's specific here is the protocol boundary.

Why the trace breaks

Distributed tracing works because each hop passes a traceparent to the next. In HTTP that's a header, and every framework's auto-instrumentation handles it without you thinking about it.

MCP breaks both halves of that assumption.

Over stdio there are no headers. A local MCP server is a subprocess exchanging JSON-RPC over stdin and stdout. There is nowhere to put a header, so header-based propagation isn't degraded — it's structurally unavailable.

Over Streamable HTTP the headers exist but describe the wrong layer. HTTP propagation covers the HTTP request, not the MCP messages inside it, and the two are independent: a retry splits one MCP call across several HTTP requests, and one response stream can carry several messages. Instrument at the HTTP layer and you get a trace of your transport, not of your work.

Both problems have the same root: the unit you want to trace is the tool call, and the transport is the wrong place to look for it.

The fix: propagate in _meta

MCP defines a reserved _meta property bag on JSON-RPC messages, and as of the 2026-07-28 specification the keys traceparent, tracestate, and baggage are reserved inside it for W3C Trace Context — an explicit exception to the usual DNS-prefix rule, carved out by SEP-414. Because _meta lives in the message body rather than the transport, it works identically over stdio and Streamable HTTP, and it's scoped to a single call.

One detail decides whether your implementation is correct: _meta sits in the request's params, alongside name and arguments — not inside the arguments. That's what lets trace context ride along without ever reaching your tool's signature.

Check your SDK before you write any of this, because it may already be done. The MCP Python SDK implements SEP-414 itself: every server emits a span per message it handles and picks up trace context from _meta automatically. What follows is what that machinery is doing, and what you write by hand when your SDK or your language doesn't.

Either way the shape is the same: inject on the client, extract on the server.

Client side — before dispatching a tool call, start a span and write the current context into _meta:

1from opentelemetry import trace
2from opentelemetry.propagate import inject
3
4tracer = trace.get_tracer("agent")
5
6
7async def call_tool(session, tool_name: str, arguments: dict):
8    # Span name follows the GenAI convention: "{operation} {target}"
9    with tracer.start_as_current_span(f"execute_tool {tool_name}") as span:
10        span.set_attribute("gen_ai.operation.name", "execute_tool")
11        span.set_attribute("gen_ai.tool.name", tool_name)
12        span.set_attribute("gen_ai.tool.type", "function")
13
14        carrier: dict[str, str] = {}
15        inject(carrier)          # traceparent, plus tracestate/baggage if set
16
17        return await session.call_tool(
18            tool_name,
19            arguments=arguments,
20            meta=carrier,        # lands in params._meta, beside arguments
21        )

Server side — extract before doing any work, and make the extracted context the parent:

1from opentelemetry import trace
2from opentelemetry.propagate import extract
3
4tracer = trace.get_tracer("mcp-server")
5
6
7async def handle_tool_call(name: str, arguments: dict, meta: dict | None):
8    parent = extract(meta or {})  # returns a Context, empty if nothing was sent
9
10    with tracer.start_as_current_span(
11        f"execute_tool {name}", context=parent
12    ) as span:
13        span.set_attribute("gen_ai.tool.name", name)
14        result = await run_tool(name, arguments)
15        span.set_attribute("gen_ai.tool.call.id", result.call_id)
16        return result

inject and extract use whatever global propagator you've configured, so this stays correct if you switch from W3C to B3 — you don't hand-parse traceparent, and you shouldn't.

One failure mode worth knowing. If extract receives an empty carrier it returns an empty context, and passing that as context= makes your server span a new root — two disconnected traces rather than an error, which looks like "tracing is broken" but is actually "propagation isn't reaching me." If you'd rather an un-propagated call nest under whatever ambient span exists, test trace.get_current_span(parent).get_span_context().is_valid and pass context=None when it's false. That's exactly what the Python SDK's own extractor does.

The span shape

Once context crosses the boundary, decide what spans to create. The GenAI conventions define an operation vocabulary; use it rather than inventing names, because the value of a convention is that a backend can build a view on top of it.

invoke_agent troubleshoot                    ← the whole task
├── chat claude-opus-5                       ← model round-trip 1
├── execute_tool get_pods                    ← MCP client span
│   └── execute_tool get_pods                ← MCP server span (via _meta)
│       └── GET /api/v1/namespaces/…/pods    ← downstream, auto-instrumented
├── chat claude-opus-5                       ← model round-trip 2
└── execute_tool get_logs
    └── execute_tool get_logs
        └── GET /api/v1/namespaces/…/log

The nested execute_tool pair is deliberate, not redundant. The outer span is what the client observed — including serialization, transport, and queueing. The inner is what the server spent. The gap between them is your transport overhead, and it's invisible if you only instrument one side.

Span names follow {gen_ai.operation.name} {target} — execute_tool get_pods, chat claude-opus-5. Keep the target low-cardinality: a tool name, not a tool name with the arguments interpolated.

Attributes worth setting

These come from the OpenTelemetry GenAI semantic conventions. Using the standard names is what makes a trace portable between backends. There's also a dedicated MCP convention covering mcp.method.name and the _meta propagation above — read that one if you're writing instrumentation rather than instrumenting your own agent.

One migration note: gen_ai.system is deprecated in favour of gen_ai.provider.name. Plenty of blog posts and older SDK versions still emit gen_ai.system, so if your dashboards group by it, they'll quietly go empty as libraries update. Grep for it.

The conventions also define gen_ai.tool.call.arguments and gen_ai.tool.call.result. Think hard before enabling those — see the pitfalls below.

What the trace actually tells you

The reason to build this is a question logs can't answer: where did the latency go?

With the span shape above, a 47-second trace decomposes immediately:

• Model latency — sum of chat spans. Fix with a smaller model, lower effort, or streaming.
• Tool latency — the inner execute_tool spans. Fix in your tool implementation; it's ordinary backend work.
• Transport latency — outer execute_tool minus inner. Usually small; when it isn't, you have a connection or serialization problem.
• Turn count — the number of chat spans. Four round-trips where you expected two means the agent is exploring, and the fix is the tool descriptions, not the infrastructure.

That last one is the most useful and least obvious. Agent latency is usually dominated by how many times it went back to the model, not by any single call being slow. A trace makes turn count visible; logs make you count by hand.

Because token counts are on the spans as attributes, the same trace also carries cost. Sum gen_ai.usage.output_tokens across a trace, multiply by your rate, and you have cost per task — and cost per tool, once you can see which tools drive extra turns. That's the API-side complement to measuring self-hosted inference cost.

Pitfalls

Don't put arguments and results in span attributes by default. gen_ai.tool.call.arguments and gen_ai.tool.call.result are the most tempting attributes in the spec and the most dangerous. A tool that reads pod logs or queries a database will put that content into your tracing backend, where it's retained, indexed, and readable by anyone with dashboard access. That's the same egress problem covered in giving an agent read-only Kubernetes access, arriving through a different door. If you need them for debugging, enable them in dev, redact in prod, and never for tools that touch user data.

Watch span size. Even without full arguments, agent traces are large — many spans, long attribute values, deep nesting. Backends bill on ingest and most enforce per-span attribute limits that truncate silently. AI workloads already inflate the observability bill more than teams expect; agent traces are a direct contributor.

Sample whole traces, never individual spans. Head sampling at the span level shreds agent traces into fragments that are worse than nothing — you get a tool span with no parent and no idea what asked for it. Sample at the root, or use tail sampling in the Collector so you can keep the slow and errored traces and drop the boring ones. For agents, "keep every trace over 10 seconds and 1% of the rest" is a good default.

Errors need record_exception and a status. A tool that fails and returns an error result to the model is not a failed span by default — the call succeeded, it just returned bad news. Set span.set_status(Status(StatusCode.ERROR)) explicitly when the tool errors, or your error rate will look implausibly good.

Where to start

If you're instrumenting an existing agent, do it in this order:

1. Model spans first. They're the easiest and usually the biggest latency contributor. You'll learn something on day one.
2. Client-side tool spans. Now you can see turn count and which tools get called.
3. _meta propagation and server spans. This is where transport overhead and slow tool internals become visible.

Steps one and two cost an afternoon and answer most latency questions. Step three is what makes the trace a distributed trace instead of a client-side timeline — worth doing once your MCP servers are real services rather than local subprocesses.

For what MCP changes about DevOps tooling more broadly, the MCP revolution in DevOps is the wider view; the Kubernetes troubleshooting agent is a concrete thing to point this at.