Kubecost: Kubernetes Cost Monitoring and Allocation

Cloud bills for Kubernetes clusters are notoriously opaque. kubectl top pods tells you resource usage — it tells you nothing about what that usage costs, which team owns it, or where the waste is. Kubecost fills that gap: it tracks every CPU, memory, storage, and network byte your workloads consume and maps those to actual dollar costs using cloud pricing data.

This tutorial covers Kubecost v3, which introduced a major architectural change from earlier versions — it no longer depends on Prometheus and uses ClickHouse for storage instead, significantly reducing the memory footprint and improving query performance.

What Kubecost Does (and Doesn't Do)

Kubecost allocates costs to Kubernetes objects — namespaces, deployments, pods, labels — based on resource requests and actual usage. It uses your cloud provider's on-demand pricing by default and can ingest actual billing exports (AWS CUR, GCP Billing, Azure Cost Management) for precise numbers that account for reserved instances, savings plans, and spot pricing.

It does not replace your cloud billing console for cross-service costs (RDS, S3, CloudFront, etc.). It's scoped to compute and storage that Kubernetes orchestrates.

Step 1: Install Kubecost with Helm

The free tier covers a single cluster with unlimited namespaces and 15 days of data retention.

helm repo add kubecost https://kubecost.github.io/kubecost/
helm repo update

helm install kubecost kubecost/kubecost \
  --namespace kubecost \
  --create-namespace

Wait for pods to be ready:

kubectl get pods -n kubecost --watch

You should see these pods reach Running:

• kubecost-frontend-* — the UI and API server
• kubecost-finops-agent-* — cost allocation engine (replaces the old cost-model)
• kubecost-*-clickhouse-* — embedded ClickHouse database for cost storage

Kubecost v3 does not bundle Prometheus. If you see Prometheus-related pods, you're likely running the legacy v2 cost-analyzer chart.

Step 2: Access the Kubecost UI

kubectl port-forward deployment/kubecost-frontend 9090 --namespace kubecost

Open http://localhost:9090. The Overview page shows total cluster spend broken down by namespace, with CPU, memory, storage, and network costs side by side.

Give it 5–10 minutes to collect initial metrics before the numbers populate.

Step 3: Connect Cloud Billing Data

Out of the box, Kubecost uses public on-demand pricing. For accurate numbers — especially if you use reserved instances, savings plans, or spot — connect your actual billing export.

AWS (Cost and Usage Report)

1. Enable a Cost and Usage Report (CUR) in the AWS Billing console. Use hourly granularity, Parquet format, delivered to an S3 bucket.

2. Create an IAM policy granting Kubecost read access to that bucket:

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Effect": "Allow",
6      "Action": ["s3:GetObject", "s3:ListBucket"],
7      "Resource": [
8        "arn:aws:s3:::your-cur-bucket",
9        "arn:aws:s3:::your-cur-bucket/*"
10      ]
11    }
12  ]
13}

3. Attach the policy to your node IAM role, or use IRSA for more granular access control.

4. Configure the CUR integration in the Kubecost UI under Settings → Cloud Integrations → AWS, or via a cloud-integration.json secret. The exact Helm values differ between versions — consult the IBM Kubecost docs for the current configuration keys.

CUR data typically appears in Kubecost within 24 hours of the first export.

GKE (GCP Billing Export)

Enable billing export to BigQuery in the GCP Console: Billing → Billing export → BigQuery export. Then configure the integration in Kubecost's UI under Settings → Cloud Integrations → GCP, providing your project ID and a service account key with BigQuery read access.

kubectl create secret generic gcp-secret \
  --from-file=service-account-json=./sa-key.json \
  -n kubecost

Step 4: Understand Cost Allocation

The Allocation page is where Kubecost earns its keep. By default it groups costs by namespace, but you can group by any Kubernetes label — which is how you get team-level or product-level cost reports.

A few groupings worth bookmarking:

By namespace — who owns what cluster real estate:

Group by: Namespace
Window: Last 7 days

By team label — requires your deployments to carry a consistent team label:

Group by: Label > team
Window: Last 30 days

By controller — per-deployment cost, useful for finding expensive single workloads:

Group by: Controller
Sort by: Total cost (desc)

The numbers Kubecost shows are allocated costs, not metered cloud costs. It distributes the node cost proportionally based on resource requests (for guaranteed resources) and actual usage (for burstable resources).

Step 5: Enforce Cost Allocation with Labels

Kubecost's allocation only works well if your workloads carry consistent labels. Without labels, everything ends up in __unallocated__ and you can't attribute cost to teams.

A minimal label convention:

1# deployment.yaml
2metadata:
3  labels:
4    app: payment-service
5    team: platform
6    env: production
7spec:
8  template:
9    metadata:
10      labels:
11        app: payment-service
12        team: platform
13        env: production

The team and env labels need to be on the pod spec, not just the deployment metadata, for Kubecost to pick them up.

Audit your current label coverage:

# Find pods without a 'team' label
kubectl get pods -A -o json | \
  jq -r '.items[] | select(.metadata.labels.team == null) | "\(.metadata.namespace)/\(.metadata.name)"'

Step 6: Set Cost Alerts

Kubecost can send alerts when a namespace or label group exceeds a cost threshold.

In the UI: Alerts → Add Alert:

• Alert type: Recurring update (weekly digest) or Spend change (% spike detection)
• Aggregation: Namespace or label
• Threshold: Dollar amount or percentage increase
• Webhook: Slack incoming webhook URL for team notifications

For a Slack alert when any namespace spends more than $500 in a week:

Type: Recurring Update
Aggregation: Namespace
Window: 7d
Threshold: $500
Webhook: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK

Kubecost posts a summary to Slack with the top spending namespaces and a link to the allocation report.

Step 7: Query the API Directly

The Kubecost API is useful for piping cost data into dashboards or custom reports. The allocation endpoint:

# Cost by namespace for the last 7 days
curl "http://localhost:9090/model/allocation?window=7d&aggregate=namespace&accumulate=true" | \
  jq '.data[0] | to_entries | sort_by(-.value.totalCost) | .[:10] | .[] | "\(.key): $\(.value.totalCost | . * 100 | round / 100)"'

Sample output:

production: $842.17
staging: $231.44
monitoring: $89.23
kube-system: $44.11

This is what you'd use to build a weekly cost report or feed into a Grafana panel alongside your performance metrics.

Common Issues

All costs show as $0.00 Kubecost hasn't pulled pricing data yet. It fetches cloud pricing on startup and again periodically. Check the finops-agent logs:

kubectl logs -n kubecost -l app=kubecost-finops-agent --tail=50 | grep -i "price\|error"

__unallocated__ is the largest cost bucket Pods are missing labels or running in system namespaces. Use the audit query from Step 5 to find unlabelled pods.

UI not loading after port-forward Confirm you're forwarding the correct deployment:

kubectl get deployments -n kubecost
kubectl port-forward deployment/kubecost-frontend 9090 -n kubecost

Pods stuck in Pending Check node resources — ClickHouse requires persistent storage. Verify a StorageClass is available:

kubectl get storageclass
kubectl describe pod -n kubecost -l app=kubecost-clickhouse

What to Do with the Data

A working Kubecost install is the start, not the end. The patterns I actually use it for:

• Weekly namespace cost digest posted to Slack via the API, so team leads know their spend without logging into a dashboard
• Pre-merge cost estimate in CI: run helm template against a branch, compare resource requests to production, estimate the cost delta
• Idle resource report: Kubecost's Efficiency page shows pods with low CPU/memory utilisation — good candidates for request downsizing or HPA configuration

The most common finding when I set this up on a new cluster: the staging environment costs more than expected because it's running at full production scale with no scheduled downtime. Kubecost makes that visible in about five minutes.

Frequently Asked Questions

Why do Kubecost's figures differ from my cloud bill?

They measure different things. Kubecost allocates based on what workloads requested and used; the bill is what the provider charged, including idle capacity, data transfer and account-level discounts. The gap between them is roughly the cost of capacity you are paying for and not using, which is often the most useful number on the page.

Does it see costs outside the cluster?

Only what you connect. Managed databases, object storage and anything else outside Kubernetes are invisible to in-cluster allocation unless billing integration surfaces them. Use it for attribution — which namespace, which team — and your provider's tooling for the total.

How long until the numbers mean something?

Allocation appears quickly but reflects only what has been observed, so early figures are not representative. Wait for a full weekly cycle before drawing conclusions, since weekend and batch patterns change the picture considerably.

What is the first thing to act on?

The gap between requests and actual usage. Over-requested workloads reserve capacity nobody uses, which inflates node count and every cost derived from it. Rightsizing requests is usually the largest available saving and needs no architectural change.

Official References

• Helm chart template guide — templates, values and the sprig function set
• Helm charts — chart structure, dependencies and hooks
• AWS Savings Plans — plan types, commitment terms and how discounts apply
• EC2 Spot Instances — interruption behaviour and capacity rebalancing