Track LLM Inference Cost in Kubernetes With OpenCost

Track LLM Inference Cost in Kubernetes With OpenCost

You can find out what your GPU nodes cost in about thirty seconds. Open the cloud console, look at eight p4d.24xlarge instances, multiply.

What you almost certainly cannot answer is: what does one million tokens cost us to serve?

That's the number that matters, because it's the only one denominated the same way as the alternative. API providers quote dollars per million tokens. Until your self-hosted number is in the same unit, "should we self-host this?" is a debate about vibes.

This post derives that number. The dollars come from OpenCost, the tokens come from vLLM, and the interesting part is joining them.

Why the standard cost dashboard doesn't answer this

Kubernetes cost allocation was designed around CPU and memory. Every tool splits a node's cost across pods in proportion to what they requested, which works well for ordinary workloads — see FinOps for Kubernetes for that general practice.

Inference breaks the model in three ways.

GPU is essentially the entire bill. On a GPU node, CPU and memory allocation is rounding error. A dashboard that splits cost by CPU request will confidently attribute your inference spend to whichever sidecar requested the most CPU. The allocation is arithmetically correct and completely useless.

Allocation is not utilization. A pod holding one A100 costs the same whether it serves ten thousand requests per minute or zero. For a web service, request volume and resource consumption move together, so allocation is a decent proxy for value delivered. For inference, they're decoupled — the GPU is reserved the moment the pod schedules, and idle time bills at exactly the same rate as saturated time.

Cost per namespace is the wrong denominator. It tells you which team spent money, not whether the money bought anything. Two teams can spend an identical $4,000 a month, one serving 400 million tokens and the other 4 million, and a namespace-level chargeback report renders them identical.

So the goal isn't a prettier allocation dashboard. It's a rate: dollars per token served.

Step 1: OpenCost, with GPU pricing configured

OpenCost is the CNCF project that Kubecost is built on. It watches the cluster, joins pod resource allocation against cloud billing rates, and exports the result to Prometheus.

helm repo add opencost https://opencost.github.io/opencost-helm-chart
helm install opencost opencost/opencost \
  --namespace opencost --create-namespace \
  --set opencost.prometheus.internal.enabled=false \
  --set opencost.prometheus.external.enabled=true \
  --set opencost.prometheus.external.url=http://prometheus-server.monitoring:80

external.enabled=true is not optional. The chart only reads external.url when that flag is set; turn off the internal Prometheus without it and OpenCost silently keeps pointing at the in-cluster default, which is a confusing way to get an empty dashboard.

On a managed cloud cluster with billing integration, GPU pricing is discovered from the provider's rate card. On anything else — on-prem, bare metal, a reserved-instance fleet whose effective rate differs from list — you have to supply it, and this is the step that quietly invalidates everyone's numbers.

OpenCost reads custom pricing from a ConfigMap. Note the shape: each pricing field is its own top-level key under data, not a JSON blob nested under a filename. OpenCost's ConfigMap watcher walks data as a flat map and sets one pricing field per key, so a stray default.json key just errors.

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: custom-pricing-model
5  namespace: opencost
6data:
7  provider: "custom"
8  description: "Effective hourly rates, not list price"
9  CPU: "0.031611"
10  RAM: "0.004237"
11  GPU: "2.48"
12  storage: "0.00005479"

custom-pricing-model is the name the Helm chart uses, and it is only consulted when you install with --set opencost.customPricing.enabled=true — that is what injects the PRICING_CONFIGMAP_NAME environment variable. Without it OpenCost looks for a ConfigMap called pricing-configs and never sees yours.

That GPU figure is dollars per GPU-hour, and it is worth getting right. If you run reserved or committed-use instances, list price can overstate your real cost by 40–60%, which is more than enough to flip a self-host-versus-API decision in the wrong direction.

Confirm OpenCost is actually emitting GPU data before building anything on top:

node_gpu_hourly_cost

If that returns nothing, the rest of this post produces zeroes. The usual causes are the GPU device plugin not being installed (see the NVIDIA GPU Operator) or the pricing model not being mounted.

Step 2: The dollars

OpenCost exports allocation and cost as separate metrics, which you multiply:

Hourly GPU spend per pod is the product of the two GPU metrics, joined on the node they share:

sum by (namespace, pod) (
  container_gpu_allocation
    * on(node) group_left()
  avg by (node) (node_gpu_hourly_cost)
)

on(node) group_left() is doing the work: many containers map to one node, so this is a many-to-one join and Prometheus rejects it without an explicit modifier. The avg by (node) on the right is not cosmetic — it collapses the cost metric to exactly one series per node, which is what OpenCost's own documented query does. Skip it and any second series for the same node (a federated or multi-cluster Prometheus, a second OpenCost replica) turns the join into a many-to-many match and the query errors out. Roll it up to a namespace by dropping pod from the by clause.

That's the conventional answer, and on its own it's the dashboard I just argued is useless. It becomes useful in the next step.

Step 3: The tokens

vLLM exposes Prometheus metrics on its /metrics endpoint, prefixed vllm:. Two counters matter here:

Both carry a model_name label identifying what that instance serves — which is what lets you split cost by model later. (vLLM's docs list these counters as vllm:prompt_tokens and vllm:generation_tokens; the _total suffix is what the Prometheus client library appends on exposition, so the names above are what you actually query.)

Tokens per hour, as a rate:

sum by (model_name) (
  rate(vllm:generation_tokens_total[1h])
) * 3600

rate() returns per-second, so multiply by 3600. Use a window at least four times your scrape interval; [1h] smooths out the burstiness that makes short windows useless for cost work.

Both are counters that reset when the pod restarts. rate() adjusts for those resets automatically, and so does increase() — it is documented as syntactic sugar for rate() multiplied by the window length, so increase(x[1h]) and rate(x[1h]) * 3600 are the same number. The reset mistake that actually bites is aggregating first: take rate() per series and sum() the result, never rate(sum(...)), which cannot see a per-pod restart at all and is a common way to get a cost figure wrong by an order of magnitude.

Step 4: The join

Now the number worth putting on a dashboard — dollars per million tokens:

1(
2  sum by (pod) (
3    container_gpu_allocation
4      * on(node) group_left()
5    avg by (node) (node_gpu_hourly_cost)
6  )
7)
8/
9(
10  sum by (pod) (
11    rate(vllm:generation_tokens_total[1h]) * 3600
12  )
13) * 1e6

Both sides are per-hour, so the hours cancel and you're left with dollars per token, scaled to a million.

The label alignment is the fiddly part. OpenCost labels by pod; vLLM's own metrics don't know they're in Kubernetes. This query assumes your Prometheus adds pod and namespace labels when scraping — which the standard kubernetes-pods service discovery relabeling does, and which the Prometheus Operator does via PodMonitor. If your scrape config doesn't, the join silently returns empty rather than erroring. Check by running each half separately and confirming the label sets actually intersect:

# Do these share a 'pod' label with the same values?
topk(5, container_gpu_allocation)
topk(5, vllm:generation_tokens_total)

If you run one model per deployment, joining by (model_name) on the token side and mapping it to namespace on the cost side is cleaner. If several models share a node, pod is the only honest join key.

Should prompt tokens count? For comparing against API pricing, keep them separate — providers bill input and output at different rates, usually 4–5× apart. Your GPU cost doesn't split that way (the same hardware does both), so the defensible comparison is total tokens against a blended API rate, or output tokens against output pricing while acknowledging you're ignoring prefill cost. Pick one, write it down, and be consistent. Silently switching denominators is how cost dashboards lose their audience.

Step 5: The number nobody wants to look at

Cost per token tells you what you're paying. This tells you what you're wasting:

1 - (
  avg by (node) (DCGM_FI_DEV_GPU_UTIL) / 100
)

DCGM_FI_DEV_GPU_UTIL comes from the DCGM exporter, which ships with the GPU Operator. Multiply that idle fraction by the node's hourly cost and you have the dollar value of GPUs you're paying for and not using.

For most self-hosted inference setups this number is uncomfortable. Inference traffic is peaky, GPUs are provisioned for peak, and the trough is dead money at full price. It's routine to find 60–70% idle on a fleet sized for a daily spike.

That's the real gap between "our GPU bill" and "our cost per token," and it's where the optimization actually lives: batching, right-sizing to a smaller model, quantization and tensor parallelism to fit more on less hardware, or scale-to-zero for workloads that tolerate cold starts. Buying fewer GPUs beats every clever allocation report.

What to do with the number

You now have a figure directly comparable to a provider's price list. Three honest caveats before you use it in a decision:

It excludes everything that isn't GPU. Load balancers, storage for weights, inter-zone traffic, and the observability bill — which for AI workloads is its own significant line item. Adding CPU and RAM allocation to the numerator via node_total_hourly_cost gets you closer.

It excludes the engineers. Someone maintains this. If self-hosting is 30% cheaper on tokens and costs a quarter of an SRE's time, it is not cheaper. This is the term most self-host business cases quietly drop.

It's only valid at your current utilization. Cost per token is inversely proportional to throughput, so the number moves every time traffic does. A quote of "$0.40 per million tokens" measured during peak becomes $1.60 during the overnight trough on the same hardware. Report it as a range across a full traffic cycle, or as a 7-day average — never as a single spot value.

With those stated, the comparison becomes a real engineering decision rather than a preference. Some workloads are clearly cheaper self-hosted: steady, high-volume, latency-tolerant traffic against a small model on hardware you already own. Others clearly aren't: spiky, low-volume traffic that leaves an A100 idle 90% of the day. When to move off Ollama for production inference covers the adjacent decision on the serving stack, and deploying an LLM on Kubernetes is the build this measures.

The point of instrumenting it is that you stop guessing which one you have.