Distributed Tracing in Microservices: From Zero to Your First End-to-End Trace

A request comes into your orders service. It calls payments. payments is slow today, so the whole request is slow — but your orders logs just show "request completed in 2.3s" and your payments logs, sitting in a different file on a different host, show nothing that obviously ties back to that one request. Logs tell you what happened inside a service. They don't tell you what happened across services for one specific call. That's the gap distributed tracing closes: every request gets a trace ID, every hop it makes gets a span tagged with that same trace ID, and a tracing backend stitches the spans back into a waterfall you can actually read.

This tutorial runs entirely on your laptop with Docker — no Kubernetes cluster, no cloud account, no service mesh. If you're running this inside Kubernetes and want the cluster-native version with Grafana dashboards, see Trace Requests End-to-End With OpenTelemetry on Tempo and Grafana instead.

What You'll Build

• A payments service (Flask) that simulates real work with time.sleep
• An orders service (Flask) that calls payments over HTTP with requests
• Both instrumented with the OpenTelemetry SDK, auto-capturing every inbound and outbound HTTP call
• Spans exported over OTLP to a local Jaeger instance running in a single Docker container
• One end-to-end trace, viewed in the Jaeger UI, showing exactly where the time went
• A manual custom span with your own attribute, for the parts auto-instrumentation can't see

Step 1: Start Jaeger

Jaeger v1 (the old jaegertracing/all-in-one image) reached end-of-life on December 31, 2025. Jaeger v2 replaces it with a single unified binary/image — still one container, still the same "collector, storage, and UI in one place" convenience, just a different image name and one extra flag. By default its OTLP receiver only listens on localhost inside the container, which makes it unreachable from the host even with the ports mapped — the --set flags below are what fix that:

1docker run -d --name jaeger \
2  -p 16686:16686 \
3  -p 4317:4317 \
4  -p 4318:4318 \
5  cr.jaegertracing.io/jaegertracing/jaeger:2.21.0 \
6  --set receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317 \
7  --set receivers.otlp.protocols.http.endpoint=0.0.0.0:4318

• 16686 — the Jaeger UI, at http://localhost:16686
• 4317 — OTLP over gRPC (what we'll use)
• 4318 — OTLP over HTTP, if you'd rather use that exporter instead
• The two --set flags rebind the OTLP receiver to 0.0.0.0 inside the container — skip them and the port mappings above do nothing, because nothing inside the container is listening on an interface the mapping can reach

Confirm it's up:

docker ps --filter name=jaeger

Open http://localhost:16686 in a browser. It'll be empty — no traces yet — but if the UI loads, Jaeger is ready to receive them.

Step 2: Install the OpenTelemetry Packages

In a fresh virtualenv:

pip install flask requests \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-grpc \
  opentelemetry-instrumentation-flask \
  opentelemetry-instrumentation-requests

The two -instrumentation-* packages are the important part: they monkey-patch Flask and requests so every inbound request and every outbound call automatically becomes a span, with no manual wrapping needed for the common case.

Step 3: The payments Service

This is the downstream service. It does a bit of fake work and returns JSON.

1# payments.py
2import time
3from flask import Flask, jsonify
4
5from opentelemetry import trace
6from opentelemetry.sdk.resources import Resource
7from opentelemetry.sdk.trace import TracerProvider
8from opentelemetry.sdk.trace.export import BatchSpanProcessor
9from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
10from opentelemetry.instrumentation.flask import FlaskInstrumentor
11from opentelemetry.instrumentation.requests import RequestsInstrumentor
12
13provider = TracerProvider(resource=Resource.create({"service.name": "payments"}))
14provider.add_span_processor(
15    BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True))
16)
17trace.set_tracer_provider(provider)
18
19app = Flask(__name__)
20FlaskInstrumentor().instrument_app(app)
21RequestsInstrumentor().instrument()
22
23@app.route("/charge", methods=["POST"])
24def charge():
25    time.sleep(0.4)  # pretend this talks to a card network
26    return jsonify({"status": "charged", "amount_cents": 1999})
27
28if __name__ == "__main__":
29    app.run(port=5001)

The Resource with service.name is what makes Jaeger show "payments" as its own service in the UI instead of an anonymous blob. Skip it and every span shows up under unknown_service.

Step 4: The orders Service

The upstream service. It receives a request and calls payments.

1# orders.py
2from flask import Flask, jsonify
3import requests
4
5from opentelemetry import trace
6from opentelemetry.sdk.resources import Resource
7from opentelemetry.sdk.trace import TracerProvider
8from opentelemetry.sdk.trace.export import BatchSpanProcessor
9from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
10from opentelemetry.instrumentation.flask import FlaskInstrumentor
11from opentelemetry.instrumentation.requests import RequestsInstrumentor
12
13provider = TracerProvider(resource=Resource.create({"service.name": "orders"}))
14provider.add_span_processor(
15    BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True))
16)
17trace.set_tracer_provider(provider)
18
19app = Flask(__name__)
20FlaskInstrumentor().instrument_app(app)
21RequestsInstrumentor().instrument()
22
23@app.route("/checkout", methods=["POST"])
24def checkout():
25    resp = requests.post("http://localhost:5001/charge")
26    return jsonify({"order": "placed", "payment": resp.json()})
27
28if __name__ == "__main__":
29    app.run(port=5000)

Notice there's no code anywhere that manually passes a trace ID from orders to payments. RequestsInstrumentor injects a traceparent header into the outgoing requests.post call, and FlaskInstrumentor on the receiving end reads that header and continues the same trace instead of starting a new one. traceparent is a W3C standard header (00-<trace-id>-<parent-span-id>-<flags>) — any OpenTelemetry-instrumented service, in any language, understands it, which is the whole point: propagation isn't tied to one framework or one vendor.

Step 5: Run It and Fire a Request

In two terminals:

python payments.py
python orders.py

Then:

curl -X POST http://localhost:5000/checkout

You should get back {"order": "placed", "payment": {"status": "charged", "amount_cents": 1999}}. Both services just emitted spans to Jaeger.

Step 6: Read the Trace

Open http://localhost:16686, pick orders from the Service dropdown, click Find Traces. You should see one trace, roughly 400ms+ long. Click into it.

You're looking at a waterfall: a top-level POST /checkout span for orders, a nested span for the outbound requests.post call, and nested inside that, a POST /charge span for payments — all under one trace ID, all lined up on a shared timeline. That's the whole value proposition in one screenshot: you can see at a glance that essentially all 400ms lives inside payments, not in orders' own logic or in network overhead.

Step 7: Add a Manual Span

Auto-instrumentation covers HTTP in and HTTP out. It has no idea what your business logic is doing in between. For that, wrap the interesting part yourself:

1tracer = trace.get_tracer(__name__)
2
3@app.route("/checkout", methods=["POST"])
4def checkout():
5    with tracer.start_as_current_span("apply-discount-rules") as span:
6        discount_cents = 200
7        span.set_attribute("discount.cents", discount_cents)
8        span.set_attribute("discount.rule", "first-order")
9
10    resp = requests.post("http://localhost:5001/charge")
11    return jsonify({"order": "placed", "payment": resp.json()})

Because this runs inside the request that Flask already instrumented, start_as_current_span automatically nests it under the active trace — you don't pass any IDs around manually. set_attribute is how you attach business context (a discount rule, a customer tier, a cart size) that becomes searchable in Jaeger later, not just a timing bar.

A Note on Sampling

Tracing every single request is fine at the traffic levels in this tutorial. It's not fine at production scale — 100% sampling means 100% of the export volume and 100% of the storage cost. The simplest starting point is head-based sampling: decide whether to keep a trace at the moment it starts, using a TraceIdRatioBased sampler:

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

provider = TracerProvider(
    resource=Resource.create({"service.name": "orders"}),
    sampler=TraceIdRatioBased(0.1),  # keep 10% of traces
)

The tradeoff: a rare, expensive error might land in the 90% you drop. Tail-based sampling (decide after seeing the whole trace, keep the slow/error ones preferentially) solves that but needs a collector sitting in front of your backend to buffer spans until a trace completes — worth knowing exists, not something to reach for on day one.

Common Issues

No traces show up in Jaeger at all. Two usual suspects. First, the exporter pointed at the wrong port — 4317 is gRPC, 4318 is HTTP, and the OTLPSpanExporter from opentelemetry-exporter-otlp-proto-grpc must use 4317. Second, if you started the container without the --set receivers.otlp.protocols.*.endpoint=0.0.0.0:... flags from Step 1, the OTLP receiver is only listening on localhost inside the container — the port mapping exists but nothing on the other end of it is reachable. Both failure modes are silent: nothing crashes, spans just vanish.

The trace breaks into two disconnected traces instead of one. This happens the moment you cross an async boundary the instrumentation doesn't automatically bridge — a message queue, a Celery task, a threading.Thread. The traceparent header only propagates automatically over HTTP calls that RequestsInstrumentor (or equivalent) wraps. Anything else, you have to extract and re-inject the trace context by hand.

Spans from a short-lived script never appear. BatchSpanProcessor batches spans and flushes on an interval (5 seconds by default) or when the batch fills up. A script that calls sys.exit() immediately after doing work can exit before the batch ever flushes. Call provider.shutdown() before exiting, or use SimpleSpanProcessor for scripts (never in a real service — it exports synchronously and will tank your latency).

Span timestamps look impossible (a child span appears to start before its parent). This is real clock skew between hosts, not a bug in your instrumentation — it's the reason distributed tracing tools generally trust span duration more than absolute wall-clock alignment across machines. Run NTP, or at minimum know it's a factor before you trust a multi-host waterfall down to the millisecond.

Frequently Asked Questions

What's the difference between a trace, a span, and a log?

A span is one unit of work with a start time, an end time, and a name — "handle POST /checkout". A trace is a tree of spans that share a trace ID, representing one end-to-end request. A log is an unstructured (or structured) timestamped message with no inherent relationship to other logs unless you manually correlate them. Traces give you the shape of a request across services for free; logs give you detail inside one span if you attach them as span events.

Jaeger vs Tempo vs Zipkin — does it matter which one I pick?

Less than it used to. All three speak OTLP now, so your instrumentation code (the part in this tutorial) doesn't change based on backend. Jaeger has the most mature standalone UI and is the easiest single-container local setup, which is why it's used here. Tempo is built to pair with Grafana and to be cheap at very high volume by skipping its own indexing. Zipkin is older and less actively developed than the other two. Pick based on where the traces need to live, not the SDK.

Do I need a service mesh (Istio, Linkerd) to get distributed tracing?

No. A service mesh can add spans for the network hops it proxies without you touching application code, which is convenient at scale, but it can't see inside your application logic — it only knows "a request came in, a request went out." The manual span in Step 7 is something a mesh alone never gives you. Application-level instrumentation and mesh-level instrumentation are complementary, not either/or.

How much overhead does this actually add?

For HTTP-level auto-instrumentation with batched, async export, the overhead per request is typically sub-millisecond to a few milliseconds — creating a span object and appending to an in-memory batch is cheap. The costs that actually bite you are indirect: exporter memory buildup if the collector is unreachable and the batch processor keeps queueing, and the storage/ingestion cost on the backend at high sampling rates. That's what Step "A Note on Sampling" above is for.

Official References

• OpenTelemetry for Python — SDK setup, exporters, and instrumentation
• OpenTelemetry Instrumentation for Flask — the auto-instrumentation package used here
• W3C Trace Context — the traceparent header spec that makes cross-service propagation vendor-neutral
• Jaeger Getting Started — the all-in-one image and its ports
• OpenTelemetry Sampling — head vs tail sampling in more depth

Next step: once you've got tracing working, pair it with metrics and logs into a single Grafana view by moving this exact setup into Trace Requests End-to-End With OpenTelemetry on Tempo and Grafana, or instrument something more realistic than a sleep call with Trace MCP Tool Calls With OpenTelemetry.