Node Problem Detector: Automated Node Health Monitoring for Kubernetes

A node can be Ready in Kubernetes' eyes and still be quietly failing — a kernel deadlock, a filesystem that just remounted read-only, a container runtime that's wedged. The kubelet's own health check doesn't look for any of this; it mostly confirms the kubelet process itself is alive and can talk to the API server. Node Problem Detector (NPD) is the Kubernetes SIG project that fills that gap — a DaemonSet that watches for node-level failure signatures and reports them as first-class Kubernetes objects.

The part that trips people up: NPD only detects and reports. It has no ability to cordon, drain, or replace a node. Deploying it alone gives you visibility and nothing else — the remediation has to come from something else entirely.

What NPD Actually Watches

NPD runs two kinds of problem daemons, both DaemonSets so every node gets its own instance:

Log monitors tail system logs (journald or syslog, depending on the node's init system) and match lines against a configured set of regex patterns — kernel panic signatures, OOM killer invocations, filesystem corruption messages, hardware-level ECC errors.

Custom plugin monitors run arbitrary scripts on an interval and interpret the exit code and output as a health signal — useful for checks NPD doesn't ship out of the box, like a custom disk-latency probe or a vendor-specific hardware diagnostic.

Both report through the same two channels:

• Node Conditions — visible via kubectl describe node, these are durable, queryable state (KernelDeadlock, ReadonlyFilesystem, FrequentKubeletRestart, FrequentContainerdRestart, CorruptDockerOverlay2)
• Kubernetes Events — transient, timestamped records of when a problem was first observed

1kubectl describe node ip-10-0-4-201.ec2.internal
2# Conditions:
3#   Type                  Status  Reason
4#   ----                  ------  ------
5#   KernelDeadlock         False   KernelHasNoDeadlock
6#   ReadonlyFilesystem     False   FilesystemIsNotReadOnly
7#   FrequentKubeletRestart False   NoFrequentKubeletRestart
8#   FrequentContainerdRestart False NoFrequentContainerdRestart
9#   Ready                  True    KubeletReady

A condition flipping to True (for the bad conditions — read them as "problem is present," not as a pass/fail in the usual Kubernetes sense) means NPD has seen the failure signature but the node is otherwise still marked Ready and will keep receiving pods unless something else acts on that condition.

Installing NPD

The upstream project ships a DaemonSet manifest, but most teams install via a community-maintained Helm chart:

helm repo add deliveryhero https://charts.deliveryhero.io/
helm repo update

helm install node-problem-detector deliveryhero/node-problem-detector \
  --namespace kube-system \
  --set hostNetwork=true

NPD needs hostPID and mounted host paths (/var/log, /dev/kmsg) to read kernel and system logs from the node it's running on — this is expected and required, not a misconfiguration to fight.

1# Values relevant to most production installs
2hostNetwork: true
3resources:
4  requests:
5    cpu: 20m
6    memory: 20Mi
7  limits:
8    cpu: 200m
9    memory: 100Mi

NPD's resource footprint is small — it's tailing logs and running periodic checks, not doing anything CPU-intensive.

Custom Plugin Monitors

Beyond the built-in kernel/log checks, NPD can run any script as a custom monitor. The plugin's exit code maps to a condition status:

1{
2  "plugin": "custom",
3  "pluginConfig": {
4    "invoke_interval": "30s",
5    "timeout": "5s",
6    "max_output_length": 80,
7    "concurrency": 3
8  },
9  "source": "disk-latency-monitor",
10  "conditions": [
11    {
12      "type": "DiskSlow",
13      "reason": "DiskLatencyIsNormal",
14      "message": "disk latency is within normal bounds"
15    }
16  ],
17  "rules": [
18    {
19      "type": "permanent",
20      "condition": "DiskSlow",
21      "reason": "DiskLatencyIsHigh",
22      "path": "/etc/npd/plugins/check_disk_latency.sh",
23      "timeout": "5s"
24    }
25  ]
26}

This is the extension point for anything vendor- or fleet-specific: a check for a known bad GPU driver version, a NIC firmware issue, or an internal disk-health tool your infrastructure team already runs outside Kubernetes.

NPD Alone Does Nothing: You Need a Remediation Layer

This is the part of NPD's design that surprises people coming from a platform like GKE, where node auto-repair is built in. On EKS or a self-managed cluster, NPD setting a KernelDeadlock condition to True changes nothing about scheduling — the node stays Ready, keeps accepting new pods, and sits there until a human notices and intervenes manually.

Draino is the tool most commonly cited for this: it watches for NPD's bad conditions and, when one appears, cordons the node (stops new scheduling) and drains it (evicts existing pods, respecting PodDisruptionBudgets) so a cluster autoscaler or Karpenter replaces it with a fresh node. Draino assumes there's no remediation for a node condition other than replacing the node entirely — it doesn't try to fix anything in place. Worth flagging before you adopt it: the original planetlabs/draino repository has seen essentially no meaningful commits since 2020-2021 — it still works, and the binary/Helm chart are stable enough that this doesn't block using it, but check for an actively maintained fork or be prepared to run it as-is with no expectation of upstream fixes.

1# Draino, configured to act on the conditions NPD reports
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: draino
6spec:
7  template:
8    spec:
9      containers:
10        - name: draino
11          image: planetlabs/draino:latest
12          args:
13            - --node-label=node-problem-detector=enabled
14            - --namespace=kube-system
15            - KernelDeadlock
16            - ReadonlyFilesystem
17            - FrequentContainerdRestart

Kured solves an adjacent but different problem — it watches for a reboot-required sentinel file (from unattended-upgrades or similar) and safely reboots nodes one at a time, respecting PodDisruptionBudgets. It's not driven by NPD conditions, but it's commonly deployed alongside NPD+Draino to cover the "needs a reboot" case that NPD's log-watching doesn't detect on its own.

On GKE, this whole layer is unnecessary for standard node problems — Google's node auto-repair already watches its own health signals and replaces unhealthy nodes automatically. NPD is still useful there for surfacing conditions into kubectl describe node and your own alerting, but the remediation loop GKE gives you for free is exactly what EKS and self-managed clusters have to assemble from NPD + Draino (or an equivalent).

Frequently Asked Questions

Does NPD replace the kubelet's own node health checks?

No, they check different things. The kubelet's own Ready/NotReady condition mostly reflects whether the kubelet process is alive and can reach the API server. NPD checks for failure modes underneath that — kernel-level problems, filesystem corruption, hardware errors — that a node can suffer while the kubelet itself keeps reporting healthy.

Will a bad NPD condition prevent new pods from scheduling onto that node?

Not by itself. NPD only sets the condition; nothing in the Kubernetes scheduler treats KernelDeadlock=True as a scheduling constraint on its own. You need something acting on the condition — a taint added via a controller watching NPD's conditions, or a remediation tool like Draino cordoning the node — to actually stop new pods from landing there.

Can I alert on NPD conditions directly with Prometheus?

Yes — kube-state-metrics exposes kube_node_status_condition for every condition on every node, NPD's included, so a standard PromQL alert on that metric works without any NPD-specific tooling: kube_node_status_condition{condition="KernelDeadlock", status="true"} == 1.

Is NPD still actively maintained?

Yes, it's an active Kubernetes SIG project under kubernetes/node-problem-detector on GitHub, with releases tracking new Kubernetes versions. The Helm chart most teams use is community-maintained rather than shipped by the SIG itself, which is normal for infrastructure add-ons in this part of the ecosystem — check the chart's own release notes against your Kubernetes version before upgrading either independently.

For the autoscaler that typically replaces nodes Draino cordons, see Kubernetes Cluster Autoscaler and Karpenter. For the PodDisruptionBudget requirements that govern how safely Draino (or any drain) can evict workloads, see Kubernetes PodDisruptionBudget and Graceful Shutdown.

Running self-managed Kubernetes and tired of finding out about bad nodes from a paging alert instead of a dashboard? Talk to us at Coding Protocols — we help platform teams build the detection-to-remediation loop that managed platforms give you for free.

Official References

• node-problem-detector (Kubernetes SIG) — DaemonSet source, built-in monitors, custom plugin format
• Draino — condition-driven cordon-and-drain remediation