Aggregate Kubernetes Logs with Loki and Grafana

Loki versus Elasticsearch is a decision you make once, and this tutorial isn't where you make it — one line is enough: Loki indexes only labels, not full text, which keeps it cheap as long as you don't fight that design. Everything below is the hands-on build: get Loki running in a mode that actually scales, get logs into it with Grafana's current collector, and query them without blowing up the index on day one.

By the end you'll have Loki deployed in simple scalable mode, Grafana Alloy tailing every pod on every node, Loki wired into Grafana as a data source, and three LogQL queries you've actually run — including one that turns log lines into a rate metric.

What You'll Build

• Loki deployed via the grafana/loki Helm chart in simple scalable deployment (SSD) mode — separate read, write, and backend targets, backed by an S3-compatible bucket
• Grafana Alloy running as a DaemonSet, discovering every pod's labels and shipping logs to Loki's write path
• A labeling scheme that keeps namespace, app, and container indexed while pod name rides along as structured metadata instead
• Loki registered as a Grafana data source, queried live in Explore with label filters, a line filter, and a metric-from-logs query
• A configured retention_period so the deployment doesn't grow chunks forever

Step 1: Add the Helm Repos and a Namespace

helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
kubectl create namespace loki

The Loki chart lives in grafana-community/helm-charts — Grafana moved it out of the original grafana/helm-charts repo in 2026. The old repo still carries a loki chart, but it's a frozen copy (chart 7.3.0) many major versions behind the community one, so installing from there appears to work and leaves you on a stale Loki; always name the repo explicitly rather than relying on whichever loki your local repo list resolves first. Alloy's chart wasn't affected by that move and still comes from grafana/helm-charts, which is why both repos get added here.

Confirm you have a cluster and a workload to point this at — any Deployment writing to stdout works; the examples below assume a pod labeled app=checkout in the default namespace.

Step 2: Deploy Loki in Simple Scalable Mode

Loki's Helm chart supports three deployment modes. Monolithic runs everything in one process — fine for a five-minute demo, wrong for anything you'd point real traffic at, because ingestion (write-heavy, bursty) and querying (read-heavy, spiky) compete for the same CPU and memory. Simple scalable deployment (SSD) splits the process into read, write, and backend targets that scale independently on a shared object store, without the operational overhead of Loki's fully microservice distributed mode. SSD is the right default the moment more than one team is writing logs. Worth knowing before you build around it: Grafana has flagged SSD for eventual deprecation in favor of monolithic (for smaller setups) or full distributed mode (at real scale) — it's still fully supported today, but don't be surprised if a future Loki major version pushes you to migrate off it.

Create loki-values.yaml. This tutorial uses the chart's bundled MinIO for object storage so the whole thing runs on a local cluster with no cloud account — swap the storage.s3 block for real S3/GCS/Azure Blob in production:

1# loki-values.yaml
2deploymentMode: SimpleScalable
3
4loki:
5  auth_enabled: false
6  commonConfig:
7    replication_factor: 1
8  schemaConfig:
9    configs:
10      - from: "2024-01-01"
11        store: tsdb
12        object_store: s3
13        schema: v13
14        index:
15          prefix: loki_index_
16          period: 24h
17  storage:
18    type: s3
19    bucketNames:
20      chunks: chunks
21      ruler: ruler
22      admin: admin
23    s3:
24      endpoint: loki-minio.loki.svc:9000
25      insecure: true
26      accessKeyId: enterprise-logs
27      secretAccessKey: supersecret
28      s3ForcePathStyle: true
29  limits_config:
30    retention_period: 336h   # 14 days
31  compactor:
32    retention_enabled: true
33    delete_request_store: s3
34
35minio:
36  enabled: true
37
38# The chart's bundled MinIO subchart is deprecated (scheduled for removal);
39# this flag is required for `helm install` to render while it's still enabled.
40ignoreMinioDeprecation: true
41
42read:
43  replicas: 2
44write:
45  replicas: 2
46backend:
47  replicas: 2
48
49gateway:
50  enabled: true

limits_config.retention_period is the whole retention story, enforced by the compactor sweeping expired chunks out of the bucket. Fourteen days here keeps the demo bucket small; in production this number is a direct cost knob — every extra day of retention is another day of chunks sitting in object storage, multiplied by every label combination you're ingesting. Set it to what your incident response and compliance needs actually require, not to "as long as possible."

helm install loki grafana-community/loki -n loki -f loki-values.yaml
kubectl -n loki rollout status statefulset/loki-write
kubectl -n loki rollout status deployment/loki-read

The chart also creates a loki-gateway Service — an nginx front door that routes both the write path (/loki/api/v1/push) and the query path (/loki/api/v1/query_range) to the right backend target. That's the single hostname everything downstream talks to.

Step 3: Ship Logs with Grafana Alloy

Alloy is Grafana's unified telemetry collector — the successor to Promtail, which is now in maintenance mode. Deploy it as a DaemonSet so one instance runs per node and pulls every container's logs through the Kubernetes API — the same mechanism kubectl logs uses, not direct filesystem access, via the loki.source.kubernetes component below:

Alloy is configured with its own HCL-inspired configuration syntax, not YAML — the Helm chart takes that config as a block of text, so write the values file first:

1# alloy-values.yaml
2controller:
3  type: daemonset
4
5alloy:
6  configMap:
7    content: |
8      discovery.kubernetes "pods" {
9        role = "pod"
10        selectors {
11          role  = "pod"
12          field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
13        }
14      }
15
16      discovery.relabel "pod_logs" {
17        targets = discovery.kubernetes.pods.targets
18
19        rule {
20          source_labels = ["__meta_kubernetes_namespace"]
21          target_label  = "namespace"
22        }
23        rule {
24          source_labels = ["__meta_kubernetes_pod_label_app"]
25          target_label  = "app"
26        }
27        rule {
28          source_labels = ["__meta_kubernetes_pod_container_name"]
29          target_label  = "container"
30        }
31        rule {
32          source_labels = ["__meta_kubernetes_pod_name"]
33          target_label  = "pod"
34        }
35      }
36
37      loki.source.kubernetes "pods" {
38        targets    = discovery.relabel.pod_logs.output
39        forward_to = [loki.process.add_structured_metadata.receiver]
40      }
41
42      loki.process "add_structured_metadata" {
43        forward_to = [loki.write.default.receiver]
44
45        // Every label on the incoming entry is seeded into the pipeline's
46        // extracted map, so "pod" is readable here. An empty value means
47        // "look up the same name as the key."
48        stage.structured_metadata {
49          values = {
50            pod = "",
51          }
52        }
53
54        // Then drop it as a label, so it stops being part of the index key.
55        stage.label_drop {
56          values = ["pod"]
57        }
58      }
59
60      loki.write "default" {
61        endpoint {
62          url = "http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push"
63        }
64      }

helm install alloy grafana/alloy -n loki -f alloy-values.yaml
kubectl -n loki rollout status daemonset/alloy

The selectors.field line matters more than it looks — discovery.kubernetes talks to the Kubernetes API, not the local node, so without a spec.nodeName field selector it discovers every pod in the cluster on every node's Alloy replica. On a 10-node cluster that's each replica tailing all ten nodes' pods through loki.source.kubernetes, meaning every log line gets ingested up to 10 times. discovery.relabel then decides which of those Kubernetes labels become Loki labels versus which get dropped or repurposed. loki.write points at the gateway Service from Step 2, so Alloy never needs to know whether Loki is monolithic or SSD underneath — it just pushes to one URL.

Step 4: Get the Labels Right

That relabel block above is the most consequential ten lines in this tutorial. Loki only indexes labels — everything else in a log line is stored as compressed, unindexed text and scanned at query time. That tradeoff is what makes Loki cheap, but only if you respect it: turning a high-cardinality value like a pod name, request ID, or user ID into an indexed label creates a new stream per unique value. A few thousand pod restarts later, the index has a few thousand near-empty streams instead of one stream with a few thousand log lines in it — this is the single most common way people blow up a Loki deployment.

The config above keeps namespace, app, and container as indexed labels — low-cardinality, bounded, exactly what you'd filter by. Pod name takes a two-step route instead: discovery.relabel promotes it to a normal label, stage.structured_metadata copies it onto each log line, and stage.label_drop then removes the label so it never reaches the index. The result is a value that's still fully queryable (| pod="checkout-7d9f8b-x2k1p") without creating a new stream per pod.

That ordering is not incidental, and getting it wrong is a silent failure rather than a config error. stage.structured_metadata's values map reads from the pipeline's extracted map, not from target labels — so pointing it at a relabel-only field like __tmp_pod_name attaches nothing at all, and the query above quietly returns no results. It works here only because Alloy seeds the extracted map with the entry's existing labels before any stage runs, which is exactly why pod has to be a real label first and dropped afterwards.

The rule of thumb: if a value's cardinality is unbounded or grows with pod churn — pod name, IP, trace ID, request ID — it belongs in the log line or as structured metadata, never as a label. If it's a small, stable set — namespace, app, environment, log level — it belongs as a label.

Step 5: Add Loki as a Grafana Data Source

If Grafana isn't already running, install it and point it at Loki with a data source provisioning file — this works whether Grafana came from the kube-prometheus-stack chart or standalone:

1# loki-datasource.yaml
2apiVersion: 1
3datasources:
4  - name: Loki
5    type: loki
6    access: proxy
7    url: http://loki-gateway.loki.svc.cluster.local
8    isDefault: false
9    jsonData:
10      maxLines: 1000

Load it as a ConfigMap using Grafana's sidecar-discovery label, so it's picked up without restarting the pod:

kubectl create configmap loki-datasource -n loki \
  --from-file=loki-datasource.yaml \
  --dry-run=client -o yaml | \
  kubectl label --local -f - grafana_datasource="1" -o yaml | \
  kubectl apply -f -

Confirm it in the UI: Connections → Data sources → Loki → Test. It should report "Data source successfully connected."

Step 6: Query with LogQL in Explore

Open Explore, select the Loki data source, and run these in order.

A basic label-filtered stream query — every log line from the checkout app in default:

{namespace="default", app="checkout"}

Add a line filter to narrow to actual errors:

{namespace="default", app="checkout"} |= "error"

|= is a substring match evaluated against the raw log line, not an indexed field — this is exactly the kind of filtering that's cheap because namespace and app already narrowed the search to one small set of streams before Loki ever scans a line.

Turn that into a metric — an error rate you could put on a dashboard or alert on:

sum(rate({namespace="default", app="checkout"} |= "error" [5m]))

rate(...[5m]) converts matching log lines into a per-second rate over a 5-minute window; sum(...) collapses it across streams (say, multiple pod replicas) into one number. Loki's dual role — log storage and a Prometheus-compatible metrics-from-logs engine — is what lets this go straight onto a Grafana panel next to your actual Prometheus metrics.

Step 7: Verify the Two Views Actually Agree

A query returning results isn't proof the pipeline is correct — it could be returning stale or partial data. Cross-check against a pod you can read directly:

kubectl logs -n default -l app=checkout --tail=50 | grep -i error

Then run the same window in Explore:

{namespace="default", app="checkout"} |= "error"

Set the time range to the last 15 minutes and compare. The lines should match — same error messages, same rough count. If Explore shows nothing but kubectl logs shows errors, check the Alloy DaemonSet first (kubectl -n loki logs -l app.kubernetes.io/name=alloy) — a relabel rule with a typo in a __meta_kubernetes_pod_label_* name silently drops the label instead of erroring, which is the most common cause of "logs are being collected but I can't find them."

Where to Go Next

• Pair this with metrics — Prometheus and Grafana on Kubernetes sets up the metrics half of the same Grafana stack, so logs and metrics live in the same dashboards.
• Alert on the error-rate query from Step 6 — SLO Burn-Rate Alerts covers turning a rate query like this one into a burn-rate alert instead of a panel you have to watch manually.
• Consider the Fluent Bit path if Alloy isn't the right fit for a non-Grafana-native stack — Kubernetes Logging with Fluent Bit and Loki covers the same destination with a different collector.
• Revisit the Loki-vs-Elasticsearch tradeoff in more depth — Loki vs Elasticsearch for Kubernetes Logging covers the decision this tutorial deliberately skipped.