Continuous Profiling on Kubernetes with Grafana Pyroscope

Logs tell you what happened. Metrics tell you something got slow. Traces tell you which request got slow and where time went between services. None of the three tell you which function burned the CPU or allocated the memory that made a single span slow in the first place — that's the question a flame graph answers, and it's why continuous profiling gets called the fourth pillar of observability. Most teams that have solid Prometheus and Loki setups, and maybe even distributed tracing, still have nothing here: profiling gets treated as a one-off tool you reach for during an incident, run locally with pprof or py-spy for five minutes, then throw away.

Grafana Pyroscope (the OSS successor to Grafana's earlier Phlare project) is built to change that by making profiling continuous and cheap enough to run in production all the time, not just when something's already on fire. This tutorial deploys Pyroscope on Kubernetes, gets profiles into it two different ways, and uses the flame graph and comparison views to actually find a hot function.

What You'll Build

• Grafana Pyroscope running on Kubernetes via the grafana/pyroscope Helm chart, in single-binary mode
• eBPF-based profiling of an entire node with zero code changes, using Grafana Alloy's pyroscope.ebpf component
• A small Python service instrumented with the pyroscope-io SDK for richer, per-function CPU and memory profiles
• Pyroscope wired up as a Grafana data source, with a flame graph open on a real workload
• A diff flame graph comparing two time windows to isolate a regression

Step 1: Deploy Pyroscope

Add the Grafana Helm repo and install Pyroscope in single-binary mode — fine for a tutorial cluster, not what you'd run for a multi-tenant production fleet, but it stands up the whole write and read path with one chart:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install pyroscope grafana/pyroscope \
  --namespace observability --create-namespace \
  --set pyroscope.structuredConfig.limits.max-query-lookback=0

Check that it came up:

kubectl -n observability get pods
kubectl -n observability rollout status statefulset/pyroscope

The single-binary chart deploys Pyroscope as a StatefulSet, not a Deployment — it wants stable identity and a persistent volume for its local block storage. rollout status deployment/pyroscope returns Error from server (NotFound), which is easy to misread as a failed install. The chart also brings up a pyroscope-alloy StatefulSet, which is the collector used for eBPF and pull-mode profiling; ignore it for now if you're pushing profiles from the SDK.

Pyroscope listens on port 4040 for both its HTTP API (where profiles get pushed) and its own minimal UI. Port-forward it so you can hit it directly while you set things up:

kubectl -n observability port-forward svc/pyroscope 4040:4040

Open http://localhost:4040 — an empty UI with no application list yet is expected; nothing has pushed a profile.

Step 2: Profile Everything With eBPF (No Code Changes)

The fastest way to get something into Pyroscope is eBPF-based profiling: it samples the stack of every process on a node using a kernel-level sampling profiler, no instrumentation, no redeploy, no language-specific agent. That's the tradeoff worth being explicit about — eBPF mode gives you whole-node CPU profiling of literally anything running, including processes you don't control, but the profiles are less detailed than SDK instrumentation: you get native/inlined stack frames resolved with varying fidelity depending on the runtime, and no memory or allocation profiles at all, only CPU.

Deploy Grafana Alloy with its pyroscope.ebpf component enabled to do the collection. eBPF profiling attaches to kernel tracepoints, which needs root and the host PID namespace — a plain DaemonSet install isn't enough:

1# alloy-ebpf-values.yaml
2controller:
3  type: daemonset
4  hostPID: true
5  volumes:
6    extra:
7      - name: kernel-debug
8        hostPath:
9          path: /sys/kernel/debug
10      - name: kernel-tracing
11        hostPath:
12          path: /sys/kernel/tracing
13
14alloy:
15  securityContext:
16    privileged: true
17  mounts:
18    varlog: true
19    extra:
20      - name: kernel-debug
21        mountPath: /sys/kernel/debug
22      - name: kernel-tracing
23        mountPath: /sys/kernel/tracing

helm install alloy grafana/alloy \
  --namespace observability \
  -f alloy-ebpf-values.yaml

Give it an Alloy config (via a ConfigMap or --set-file) that scrapes the node with eBPF and pushes to Pyroscope:

1// alloy-ebpf.alloy
2discovery.process "all" { }
3
4pyroscope.ebpf "node" {
5  targets    = discovery.process.all.targets
6  forward_to = [pyroscope.write.local.receiver]
7}
8
9pyroscope.write "local" {
10  endpoint {
11    url = "http://pyroscope.observability.svc.cluster.local:4040"
12  }
13}

pyroscope.ebpf doesn't scrape anything itself and doesn't expose targets for a downstream component to pull from — it pushes profiles for whatever discovery.process hands it directly to forward_to. discovery.process "all" enumerates every process on the node; that's the "profile everything" part.

Apply it as the Alloy config and restart the DaemonSet, then check Pyroscope's UI again — you should see an application name like process.cpu per node show up within a minute or two, with every process on that node contributing samples.

This is the right default when the goal is "profile everything with no app changes" — new nodes and new processes are covered automatically. It's the wrong tool when you need memory profiles or fine-grained per-function detail for one specific service; that's what SDK instrumentation is for.

Step 3: Profile One Service With the Python SDK

For a service you control, language-native instrumentation gives you sharper, function-level detail and — critically — memory/allocation profiles, which eBPF mode can't produce. Install the SDK in your app:

pip install pyroscope-io

Start the profiler at process startup, tagged with the pod and service name so you can filter to exactly this workload in the UI later:

1# app.py
2import os
3import pyroscope
4
5pyroscope.configure(
6    application_name="checkout-service",
7    server_address="http://pyroscope.observability.svc.cluster.local:4040",
8    tags={
9        "pod": os.environ.get("HOSTNAME", "unknown"),
10        "env": os.environ.get("ENVIRONMENT", "dev"),
11    },
12)
13
14# rest of your app — the profiler now samples in the background

Deploy this as a normal Kubernetes Deployment, pointing server_address at the in-cluster Pyroscope service (no need to port-forward from inside the cluster). Within a minute of the pod starting, checkout-service shows up as a separate application in Pyroscope alongside the eBPF process.cpu profiles — same backend, two collection paths, filterable independently by the pod and env tags you attached.

Step 4: Add Pyroscope as a Grafana Data Source

If you already have Grafana running for the metrics/logs/traces side of your stack, point it at Pyroscope rather than using Pyroscope's built-in UI day to day — you get flame graphs alongside the dashboards you already have.

In Grafana: Connections → Data sources → Add data source → Pyroscope, and set the URL to the in-cluster service:

http://pyroscope.observability.svc.cluster.local:4040

Save and test, then open Explore, pick the Pyroscope data source, and select checkout-service (or process.cpu for the eBPF profiles) from the profile type / application dropdown. You'll land on a flame graph for the selected time range.

Reading a flame graph

This trips people up the first time: width represents time (samples), not call order. Each row is a level of the call stack — the root frame at top, callees below it — and a frame's width is proportional to how much of the total sampled time was spent in that function and everything it called. Wider is hotter. A frame's horizontal position has no meaning relative to its siblings; two boxes side by side are not "before" and "after" each other in execution order, they're just two different call paths that both got sampled. To find the actual hot function rather than a hot call path, look for the widest frame near the bottom of the stack (the leaf) — that's where the sampler actually caught the CPU executing, as opposed to a wide-but-shallow parent that's wide only because everything underneath it is.

Step 5: Compare Two Time Windows With the Diff View

A single flame graph tells you where time went right now. The comparison / diff view is what turns "the CPU graph went up" into "this specific function is why" — it renders two flame graphs side by side (or as a single diff, colored red/green for regressed/improved) for two different time ranges of the same application.

In Grafana Explore with the Pyroscope data source, switch to Comparison mode, then set:

• Baseline window — e.g. the 10 minutes before a deploy, or before a load test started
• Comparison window — e.g. the 10 minutes after the deploy, or during the load spike

Frames that grew between the two windows show up red and wider; frames that shrank show up green. Instead of eyeballing two flame graphs and guessing, the diff highlights exactly which function's share of total CPU time changed — this is the step that lets you say "the new serialization path in encode_response is 40% of CPU now, it was 8% before the deploy" instead of "latency went up after we shipped."

Step 6: Understand the Overhead

Continuous profiling only earns its "continuous" name if it's cheap enough to leave on. Sampling-based profilers like Pyroscope's SDK and eBPF collectors are typically in the 2–5% CPU overhead range for default sample rates — treat that as an order-of-magnitude range, not a guarantee, since actual overhead depends heavily on your sample rate, language runtime, and how allocation-heavy the workload is. At that level it's reasonable to run both eBPF node-wide profiling and SDK instrumentation on your hottest services continuously in production, the same way you'd leave Prometheus scraping on all the time. If you push the sample rate up specifically to chase a hard-to-reproduce issue, treat that higher rate as investigation-only and dial it back down once you've got what you need — that's the only overhead tradeoff worth being careful about here.

Step 7: Verify It Actually Works

Prove the setup surfaces a real hot path, not just noise. Add an obviously wasteful function to the Python service from Step 3 and hit it a few times:

@app.route("/slow")
def slow_endpoint():
    total = 0
    for _ in range(50_000_000):  # tight loop, deliberately wasteful
        total += 1
    return {"total": total}

Generate some load against it:

for i in $(seq 1 20); do curl -s http://checkout-service/slow > /dev/null; done

Open the flame graph for checkout-service in Grafana Explore over the last 5 minutes. slow_endpoint should appear as the widest leaf frame in the stack — confirming the profiler correctly attributes CPU time to the function actually burning it, not to a generic "request handler" frame. If you see that, the pipeline from SDK to Pyroscope to Grafana is working end to end.

Where to Go Next

Profiling closes the loop the other three pillars leave open — it tells you which function, not just which request or which service. From here:

• Pair it with the metrics pillar — Prometheus and Grafana on Kubernetes is the CPU/memory dashboard that tells you when to go look at a flame graph.
• Pair it with the logs pillar — Aggregate Kubernetes Logs with Loki and Grafana covers the same single-binary Helm deployment pattern used here, for logs instead of profiles.
• Pair it with the traces pillar — Distributed Tracing on Kubernetes with Tempo and Grafana finds the slow span; this tutorial finds the slow function inside it.
• Use it for right-sizing — Kubernetes Cost Optimization and FinOps covers trimming CPU/memory requests; a flame graph is usually how you find the specific function driving an oversized request in the first place.