Chaos Engineering on Kubernetes: LitmusChaos, Chaos Mesh, and Why Killing Pods Teaches You Nothing

Chaos engineering is running a controlled experiment against a stated hypothesis. It is not randomly deleting pods. The distinction matters because the random-pod-deletion version is what most teams actually implement, and it teaches them almost nothing.

Think about what you learn when you delete a pod on Kubernetes. If the service stays up: you learned that a ReplicaSet creates a replacement, which is the entire premise of the platform. If the service goes down: you did not discover a resilience weakness — you discovered you are running a single replica, or you have no PodDisruptionBudget, or your readiness probe is wrong. All three are visible by reading the manifest.

Real chaos engineering targets the failures your architecture has opinions about and your tests do not cover: a dependency that gets slow rather than dying, a DNS resolver that intermittently fails, a node that runs out of memory, a network partition where both sides think they are healthy.

The discipline before the tool

An experiment has four parts, and skipping any of them turns it into an outage you caused on purpose.

1. A steady-state definition. A measurable indicator of normal, expressed in business or user terms — checkout success rate above 99.5%, p99 latency under 400 ms. Not "CPU looks fine." If you cannot state your steady state numerically, stop here and go set up SLOs and error budgets first. Chaos engineering without SLOs is just breaking things.

2. A hypothesis. Written down, before you start. "When the payments service p99 latency rises to 3 seconds, checkout success stays above 99.5% because the circuit breaker opens after 2 seconds and we fall back to deferred capture." That is a hypothesis: specific, falsifiable, and naming the mechanism you believe protects you.

3. A blast radius. What is the smallest injection that tests the hypothesis? One pod, one namespace, 5% of traffic, one availability zone. Start smaller than feels useful.

4. An abort condition. A metric threshold that stops the experiment automatically. Decided in advance, because judgement is unreliable when a graph is heading downward and people are watching.

The single most valuable output of chaos engineering is often not a fixed bug. It is discovering that you could not tell what was happening — that your dashboards did not show the fault, or your alerts fired for the wrong service, or nobody knew which team owned the failing dependency.

The failures actually worth injecting

Ranked roughly by how often they find something real:

Latency in a dependency. The highest-yield injection by a wide margin. Services handle dependencies that die reasonably well — connections fail fast, errors propagate. Services handle dependencies that get slow very badly: connection pools exhaust, threads block, timeouts cascade, and a slow database takes down services that never touch it. Almost every large outage has this shape.

DNS failure and delay. CoreDNS is a shared dependency of everything. Intermittent resolution failure produces some of the most confusing incidents in Kubernetes, and very little application code handles it gracefully. See CoreDNS in production.

Partial and intermittent failure. A dependency failing 10% of requests is far harder to survive than one failing 100%. Retry logic amplifies it, and retry storms turn a degradation into an outage.

Resource exhaustion. Memory pressure on a node, CPU throttling, disk filling. This tests whether your requests and limits are honest — see requests, limits and QoS — and whether the right pods get evicted.

Zone loss. Do your replicas actually span zones? Topology spread constraints and pod anti-affinity are frequently declared and rarely verified. This is the experiment that most often finds a genuine surprise.

Clock skew. Rare, but brutal when it bites: certificate validation, token expiry, and distributed locks all assume clocks roughly agree.

LitmusChaos

Litmus is a CNCF project that models chaos as Kubernetes resources under litmuschaos.io/v1alpha1:

The separation is the useful part: a platform team curates ChaosExperiment definitions, and application teams create ChaosEngine objects that reference them. Experiment definitions are governed centrally; running them is self-service.

A real ChaosEngine:

1apiVersion: litmuschaos.io/v1alpha1
2kind: ChaosEngine
3metadata:
4  name: nginx-chaos
5  namespace: default
6spec:
7  appinfo:
8    appns: ''
9    applabel: ''
10    appkind: ''
11  # It can be active/stop
12  engineState: 'active'
13  chaosServiceAccount: pod-delete-sa
14  experiments:
15    - name: pod-delete
16      spec:
17        components:
18          env:
19            # set chaos duration (in sec) as desired
20            - name: TOTAL_CHAOS_DURATION
21              value: '30'
22
23            # set chaos interval (in sec) as desired
24            - name: CHAOS_INTERVAL
25              value: '10'
26
27            # pod failures without '--force' & default terminationGracePeriodSeconds
28            - name: FORCE
29              value: 'false'
30
31             ## percentage of total pods to target
32            - name: PODS_AFFECTED_PERC
33              value: ''

Points worth noting:

• engineState: 'active' — set it to stop to halt an experiment. Your abort procedure is a kubectl patch, which is worth rehearsing before you need it at speed.
• chaosServiceAccount — each experiment runs under a service account you provide, scoped to what that fault needs. Litmus does not hand itself cluster-admin, and you should not either.
• PODS_AFFECTED_PERC — the blast radius control. Set it explicitly. Empty means the experiment default, which may be more than you intended.
• FORCE: 'false' — deletes respecting terminationGracePeriodSeconds, which tests graceful shutdown. Setting true tests ungraceful termination. These are different experiments and both are worth running; see PodDisruptionBudgets and graceful shutdown.

Probes are what make Litmus an experiment framework rather than a fault injector. A probe runs during chaos — an HTTP check, a PromQL query, a command — and its result determines the verdict recorded in ChaosResult. This is where your steady-state hypothesis becomes machine-checked: assert that checkout success stays above 99.5% while the fault is active, and the experiment passes or fails on its own. Without probes you are injecting faults and eyeballing dashboards.

Chaos Mesh

Chaos Mesh, also CNCF, takes the opposite modelling approach: a distinct CRD per fault category under chaos-mesh.org/v1alpha1.

PodChaos, NetworkChaos, StressChaos, IOChaos, DNSChaos, HTTPChaos, TimeChaos, KernelChaos, JVMChaos, BlockChaos, plus cloud-level faults (AWSChaos, GCPChaos, AzureChaos), PhysicalMachineChaos for non-Kubernetes targets, and Schedule, Workflow and StatusCheck for orchestration.

1apiVersion: chaos-mesh.org/v1alpha1
2kind: PodChaos
3metadata:
4  name: pod-kill-example
5spec:
6  action: pod-kill
7  mode: one
8  selector:
9    labelSelectors:
10      "app.kubernetes.io/component": "tikv"

Compare this to the Litmus manifest. Chaos Mesh is terser and the fault type is the kind, so kubectl get networkchaos tells you what network faults exist. mode is the blast radius (one, all, fixed, fixed-percent, random-max-percent) and selector targets by label, namespace, annotation, or field.

Chaos Mesh's network and I/O fault injection is the more sophisticated of the two — latency, packet loss, corruption, duplication, bandwidth limits, and partitions, with direction control. Given that dependency latency is the highest-yield experiment, that matters.

Choosing between them

Pick Litmus if you want a governed platform where a central team curates experiments, teams self-serve, and results are recorded with pass/fail verdicts. The probe model is its real advantage.

Pick Chaos Mesh if you want precise fault injection — particularly network and I/O — and are happy to assert steady state from your own monitoring. The CRD-per-fault design is more discoverable and composes better with GitOps.

Both are CNCF projects in active development. This is not a decision worth long deliberation; the discipline matters more than the tool.

Prerequisites: do these first

Running chaos experiments before this list is complete produces outages, not learning.

1. Observability that would actually show the fault. If you inject 3 seconds of latency into a dependency and your dashboards look unchanged, you have found something — but you cannot run any further experiments until it is fixed. Distributed tracing is close to mandatory; see OpenTelemetry on Kubernetes.

2. Defined SLOs. Your steady-state hypothesis needs a number. Without one, "did the experiment pass?" is a matter of opinion.

3. PodDisruptionBudgets and multiple replicas. If a single pod deletion causes an outage, you do not need chaos engineering — you need a second replica. Fix the obvious things first; chaos engineering is for finding the non-obvious ones.

4. An incident process. Experiments occasionally become incidents. Everyone should know that a chaos experiment is running, and the abort procedure should be practised. See incident management and runbooks.

Running it without causing an incident

Start in staging, but know its limits. Staging finds bugs in your code's failure handling. It cannot find the ones that depend on production traffic volume, real data distribution, or the actual dependency graph — which is where the interesting failures live. Staging is a stage, not a destination.

Game days before automation. Schedule the first experiments, announce them, have the owning team watching. The value of the first several runs is as much organisational as technical: who noticed, how fast, did they know what to do.

Automate only what has passed manually. Once an experiment has run cleanly several times, put it on a schedule as a regression test. An experiment that used to pass and now fails is a genuine signal — something changed.

Never automate an experiment you have not run manually. This is how chaos engineering earns its bad reputation.

Keep the blast radius small for longer than feels necessary. The instinct after three successful runs is to scale up. Resist it; the failures you are hunting are non-linear, and 50% is not 5% five times bigger.

Frequently Asked Questions

Is chaos engineering just randomly breaking things in production?

No, and that framing is why it gets resisted. It is a controlled experiment with a written hypothesis, a defined steady state, a limited blast radius, and an automatic abort condition. Randomly breaking things without those four elements is an outage you scheduled.

Do I need to run it in production?

Eventually, if you want the full value. Many real failure modes only appear under production traffic volume, real data, and the actual dependency graph. But production is where you go after the experiment runs cleanly in staging several times, with a small blast radius and a rehearsed abort. Teams that start in production do it once and never again.

What should my first experiment be?

Not pod deletion. Inject 2–3 seconds of latency into one non-critical dependency of one service, at the smallest blast radius your tool allows, and watch whether your dashboards show it and your alerts fire correctly. You are testing your observability first — everything else depends on it.

LitmusChaos or Chaos Mesh?

Litmus if you want a governed platform with built-in probes that turn steady-state hypotheses into automatic pass/fail verdicts. Chaos Mesh if you want precise fault injection, especially network and I/O, and will assert steady state from your own monitoring. Both are CNCF projects; the difference matters far less than whether you run disciplined experiments.

How do I stop an experiment quickly?

In Litmus, patch the ChaosEngine to engineState: 'stop'. In Chaos Mesh, delete the chaos resource. Whichever you use, rehearse it before your first real experiment — including confirming that whoever is on call knows the command without looking it up.

Won't this cause outages?

It can, which is why the blast radius and abort conditions are not optional. The honest framing for stakeholders is that these failures will happen anyway, at 3 a.m., unannounced, with nobody prepared. A chaos experiment is the same failure at 2 p.m. on a Tuesday with the right people watching and a stop button.

Does Kubernetes make chaos engineering unnecessary?

The opposite. Kubernetes handles simple failures — a dead pod, a dead node — so well that teams assume they are resilient. What it does not handle is a dependency that is slow, a network that is intermittent, or an application that mishandles a restart. Those failures are now the ones that cause outages, and they are exactly what chaos experiments target.

See also

• SLOs, Error Budgets and Burn-Rate Alerts — the steady state your hypothesis needs
• SRE Incident Management and Runbooks — what to have in place before experimenting
• PodDisruptionBudgets and Graceful Shutdown — fix these before injecting faults
• OpenTelemetry Kubernetes Observability — seeing the fault when you inject it
• Kubernetes Debugging Guide — for when an experiment finds something