Chaos Engineering on Kubernetes with Chaos Mesh

Chaos engineering means running a controlled experiment against a hypothesis, not randomly deleting pods — that framing, plus a comparison of failure modes actually worth injecting, is covered in Chaos Engineering on Kubernetes with LitmusChaos. This tutorial skips the theory and goes straight to mechanics: installing Chaos Mesh, a CNCF chaos engineering platform for Kubernetes, and running three experiment types against a real Deployment so you can see what each one actually does to a running app.

Chaos Mesh and LitmusChaos solve the same problem with different shapes — Chaos Mesh leans on CRDs and a dashboard with a built-in pause/abort control, which is why it's worth a dedicated walkthrough even if you've already read the Litmus comparison.

What You'll Build

• Chaos Mesh installed via Helm into its own chaos-mesh namespace
• A 3-replica sample Deployment and Service to use as the experiment target
• A PodChaos experiment that kills a pod and lets you watch the Deployment self-heal
• A NetworkChaos experiment that injects latency and shows the effect with timed curl requests
• A StressChaos experiment that loads a pod's CPU and shows how resource limits and HPA respond
• A recurring Schedule that runs pod-kill chaos on a cadence instead of one-shot

Step 1: Install Chaos Mesh

Add the Helm repo and install into a dedicated namespace — keeping chaos tooling out of your app namespaces makes it trivial to kubectl delete namespace chaos-mesh and remove every trace of it later.

1kubectl create namespace chaos-mesh
2
3helm repo add chaos-mesh https://charts.chaos-mesh.org
4helm repo update
5
6helm install chaos-mesh chaos-mesh/chaos-mesh \
7  --namespace chaos-mesh \
8  --set chaosDaemon.runtime=containerd \
9  --set chaosDaemon.socketPath=/run/containerd/containerd.sock \
10  --version 2.8.4

The chaosDaemon.runtime/socketPath flags matter — Chaos Mesh's daemon talks to the container runtime directly to do things like inject network faults at the network-namespace level, so it needs to know which socket to use. If you're on kind, this is containerd at the path above; on Docker Desktop or a managed cluster, check kubectl -n chaos-mesh logs -l app.kubernetes.io/component=chaos-daemon for connection errors and adjust.

Three components come up:

• chaos-controller-manager — watches the chaos CRDs (PodChaos, NetworkChaos, etc.) and reconciles them, similar to any other Kubernetes controller
• chaos-daemon — a DaemonSet with elevated privileges that does the actual fault injection (killing processes, manipulating iptables/tc for network faults, injecting stress into cgroups) — this is why the runtime socket has to be right
• dashboard — a web UI for building, running, and — critically — pausing or aborting experiments without a terminal

kubectl -n chaos-mesh get pods

Wait until chaos-controller-manager, chaos-daemon-* (one per node), and chaos-dashboard are Running.

Port-forward the dashboard so you have it open for Step 7:

kubectl -n chaos-mesh port-forward svc/chaos-dashboard 2333:2333

Open http://localhost:2333. The dashboard runs with RBAC-scoped login enabled by default, so it won't just let you in — use the login screen's built-in Token Generator to produce an RBAC manifest, apply it, then read the token out of the Secret it creates:

# The dashboard's Token Generator UI writes an RBAC YAML like this one
# for a cluster-manager role — apply it, then fetch the resulting token:
kubectl apply -f rbac.yaml
kubectl describe -n chaos-mesh secrets $(kubectl -n chaos-mesh get secrets -o name | grep account-chaos-mesh-manager)
# on a cluster new enough to support the TokenRequest API, this also works:
kubectl create token -n chaos-mesh account-chaos-mesh-manager

Paste the token into the login screen.

Step 2: Deploy the Target App

Chaos Mesh needs something to break. Deploy a small 3-replica app with a Service in front of it:

1# target-app.yaml
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: demo-app
6  namespace: default
7spec:
8  replicas: 3
9  selector:
10    matchLabels: { app: demo-app }
11  template:
12    metadata:
13      labels: { app: demo-app }
14    spec:
15      containers:
16        - name: demo-app
17          image: hashicorp/http-echo:1.0
18          args: ["-text=hello from demo-app", "-listen=:5678"]
19          ports:
20            - containerPort: 5678
21          resources:
22            requests: { cpu: 100m, memory: 64Mi }
23            limits: { cpu: 500m, memory: 128Mi }
24---
25apiVersion: v1
26kind: Service
27metadata:
28  name: demo-app
29  namespace: default
30spec:
31  selector: { app: demo-app }
32  ports:
33    - port: 80
34      targetPort: 5678

kubectl apply -f target-app.yaml
kubectl rollout status deployment/demo-app

Confirm it's reachable before you start breaking it:

kubectl run curl-test --rm -it --image=curlimages/curl --restart=Never -- \
  curl -s http://demo-app.default.svc.cluster.local

Step 3: PodChaos — Kill a Pod and Watch It Self-Heal

This is the baseline experiment: does the ReplicaSet controller actually replace a killed pod, and how fast? Open a second terminal first so you can watch pods change live:

kubectl get pods -l app=demo-app -w

In your first terminal, apply the experiment:

1# podchaos-kill.yaml
2apiVersion: chaos-mesh.org/v1alpha1
3kind: PodChaos
4metadata:
5  name: demo-app-pod-kill
6  namespace: default
7spec:
8  action: pod-kill
9  mode: one
10  selector:
11    namespaces:
12      - default
13    labelSelectors:
14      app: demo-app

kubectl apply -f podchaos-kill.yaml

mode: one picks a single random pod matching the selector; action: pod-kill sends a delete straight to the API server (as opposed to pod-failure, which taints the pod so it stays gone for a duration instead of being immediately replaced). In the watch terminal you should see one demo-app-* pod go to Terminating, then a new one appear and reach Running within a few seconds — that's the ReplicaSet doing its job, and it's the same behavior LitmusChaos's pod-delete experiment exercises, just via a different CRD shape.

PodChaos with action: pod-kill is a one-shot experiment — it fires once and finishes. Confirm that:

kubectl get podchaos demo-app-pod-kill

Step 4: NetworkChaos — Inject Latency

Kill tests validate the control loop; latency tests validate the app's timeout and retry behavior, which is a much more common real-world failure than a pod vanishing outright.

Time a baseline request first:

kubectl run curl-test --rm -it --image=curlimages/curl --restart=Never -- \
  curl -s -o /dev/null -w "time_total: %{time_total}s\n" http://demo-app.default.svc.cluster.local

Now apply 200ms of delay to traffic from demo-app pods:

1# networkchaos-delay.yaml
2apiVersion: chaos-mesh.org/v1alpha1
3kind: NetworkChaos
4metadata:
5  name: demo-app-latency
6  namespace: default
7spec:
8  action: delay
9  mode: all
10  selector:
11    namespaces:
12      - default
13    labelSelectors:
14      app: demo-app
15  delay:
16    latency: "200ms"
17    jitter: "10ms"
18  duration: "5m"

kubectl apply -f networkchaos-delay.yaml

mode: all applies the fault to every pod matching the selector rather than a random one — for a latency test you usually want the whole tier affected, not one lucky pod skewing your measurement. Re-run the timed curl from above; time_total should now sit around 200ms higher (Chaos Mesh injects the delay via tc on the pod's network namespace, so it's real network-layer latency, not simulated at the app level). If your app calls a downstream dependency rather than serving directly, point the curl at that call path instead — the visible symptom you're after is the same either way: request latency (or timeout errors, if 200ms trips a client timeout) that traces back cleanly to this one experiment.

Let it run its 5-minute duration or delete it early to remove the fault immediately:

kubectl delete networkchaos demo-app-latency

Step 5: StressChaos — Load CPU and Watch Limits Respond

StressChaos runs the stress-ng tool inside the target container's own cgroup and namespaces, so the load is real and enforced by whatever CPU/memory limits you set on the container. No sidecar is added and the pod is not restarted — the chaos-daemon DaemonSet on the node joins the existing container and spawns the stressor there, so kubectl get pod shows the same container count and the same age as before. Don't go looking for a new container as confirmation the experiment applied; check the StressChaos object's status, or watch the pod's CPU usage, instead.

1# stresschaos-cpu.yaml
2apiVersion: chaos-mesh.org/v1alpha1
3kind: StressChaos
4metadata:
5  name: demo-app-cpu-stress
6  namespace: default
7spec:
8  mode: one
9  selector:
10    namespaces:
11      - default
12    labelSelectors:
13      app: demo-app
14  stressors:
15    cpu:
16      workers: 2
17      load: 100
18  duration: "3m"

kubectl apply -f stresschaos-cpu.yaml

stressors.cpu.workers: 2 spins up two CPU-bound worker processes at load: 100 (100% of a core each) inside the target pod. Because demo-app has resources.limits.cpu: 500m (Step 2), watch the pod's actual CPU usage get capped there rather than climbing unbounded:

kubectl top pod -l app=demo-app

kubectl top needs the metrics-server add-on, which neither kind nor a fresh minikube installs by default. On minikube: minikube addons enable metrics-server. On kind, install it manually and add --kubelet-insecure-tls to its args (kind's kubelet certs aren't signed for this by default) — see the metrics-server install docs if you don't have it running yet. Without it, watch kubectl describe pod -l app=demo-app instead and look for the CPU throttling behavior in events, or use the Chaos Dashboard's own resource graphs.

If this Deployment had a HorizontalPodAutoscaler targeting CPU utilization, this is the experiment that would trigger it — the stress load pushes utilization above the target threshold, the HPA controller notices on its next sync, and you'd see kubectl get hpa -w add replicas within a minute or two. HPA with Custom Metrics covers wiring an HPA up if you want to extend this experiment that way; without one, the CPU limit is your only backstop, and this experiment is exactly how you'd confirm it's set correctly.

Step 6: Schedule — Recurring Chaos Instead of One-Shot

A single pod-kill proves resilience once. A Schedule proves it stays true as the app changes — it wraps any of the experiment types above in a cron-style cadence:

1# schedule-pod-kill.yaml
2apiVersion: chaos-mesh.org/v1alpha1
3kind: Schedule
4metadata:
5  name: demo-app-recurring-kill
6  namespace: default
7spec:
8  schedule: "*/15 * * * *"
9  historyLimit: 5
10  concurrencyPolicy: Forbid
11  type: PodChaos
12  podChaos:
13    action: pod-kill
14    mode: one
15    selector:
16      namespaces:
17        - default
18      labelSelectors:
19        app: demo-app

kubectl apply -f schedule-pod-kill.yaml

schedule is a standard cron expression — every 15 minutes here. concurrencyPolicy: Forbid skips a run if the previous one hasn't finished, which matters once you have longer-running experiment types in a Schedule. historyLimit caps how many past runs Chaos Mesh keeps around for inspection. This is the shape you'd actually run in a staging environment long-term — continuous low-grade chaos that a dashboard like SLO Burn-Rate Alerts should catch if it ever produces a real regression.

Clean it up once you're done, since a forgotten Schedule keeps killing pods indefinitely:

kubectl delete schedule demo-app-recurring-kill

Step 7: Safety — Scope, Environment, and the Kill Switch

Three things worth internalizing before you point any of this at something that matters:

• Scope every experiment with selector.namespaces. Every YAML above sets it explicitly to default — without it, a broad label selector like app: demo-app could match a same-named app in another namespace you didn't intend to touch.
• Run this in a non-prod cluster first, always. kind/minikube for learning the CRDs, a staging cluster with production-like traffic before anything touches prod.
• The dashboard can pause or abort a running experiment. This matters specifically because a bad experiment outlives the terminal session that started it — a NetworkChaos with a long duration keeps injecting latency whether or not you're still watching. From the dashboard's Experiments view, select the running experiment and use Pause (suspends the fault, resumable) or the delete action (removes it and reverts the fault immediately). The same is available from the CLI with kubectl delete <kind> <name> — you don't strictly need the dashboard open, but it's the faster path when you're mid-incident and don't want to remember exact resource names.

Step 8: Verify the Result

Check an experiment's status directly rather than assuming it ran:

kubectl get podchaos
kubectl describe podchaos demo-app-pod-kill

describe shows a Status section with Phase (e.g. Finished for a one-shot PodChaos, Running for anything with a duration still in progress) and Conditions reporting whether the injection actually succeeded — a Selected condition of False typically means your labelSelectors matched nothing, which is worth catching immediately rather than assuming the silence meant success.

Reading the outcome correctly matters as much as running the experiment:

• Handled cleanly: pod-kill — replacement pod Running within a few seconds and no error rate spike in your app's dashboards; latency injection — request latency rises by roughly the injected delay and then returns to baseline the moment the experiment ends, no cascading timeouts elsewhere; CPU stress — usage plateaus at the container's limits.cpu and the app keeps serving, just slower.
• The app didn't handle this well: pod-kill — requests error out for longer than a few seconds (check readiness probes and whether you have enough replicas), or the pod doesn't get replaced at all (check the Deployment isn't paused or blocked by a quota); latency injection — a downstream client throws hard errors instead of degrading (missing or too-tight timeout), or the effect never clears after you delete the NetworkChaos (check tc qdisc show on the node — a daemon crash mid-experiment can occasionally leave rules behind); CPU stress — the whole node's other workloads slow down too, meaning your limits.cpu either isn't set or isn't being enforced.

Either result is useful. A clean pass is confirmation; a bad one is a finding — exactly the kind Kubernetes Debugging and Troubleshooting Guide walks through diagnosing once you need to trace an experiment's blast radius past what describe shows you.

Where to Go Next

• Compare against LitmusChaos — Chaos Engineering on Kubernetes with LitmusChaos covers the same experiment classes on a different platform, plus the hypothesis-driven framing this tutorial skipped.
• Wire up autoscaling to react to real load — HPA with Custom Metrics extends the StressChaos experiment in Step 5 into an actual scale-out test.
• Make sure something is watching — SLO Burn-Rate Alerts is the alerting layer that should fire when a Schedule from Step 6 finds a real regression.
• Diagnose a bad blast radius — Kubernetes Debugging and Troubleshooting Guide is where to go when an experiment's effect spreads further than the selector suggested it should.