How to Taint a Node in Kubernetes — and When You Actually Should

Tainting a node is a single kubectl command — the hard part is choosing the right effect and knowing when a taint is the wrong tool entirely.

kubectl taint nodes node-1 dedicated=gpu:NoSchedule

That command makes node-1 repel every pod that does not explicitly tolerate the taint. Nothing about the pods changed; the node now rejects them by default. That inversion — the node opting out of workloads, rather than workloads opting into the node — is the whole idea, and it is exactly what makes taints different from nodeSelector and node affinity.

This post covers the anatomy of a taint, the three effects and how they actually differ, the toleration YAML that matches them, when to reach for taints versus affinity, and the real-world patterns where taints earn their keep.

Taint anatomy: key=value:effect

A taint is three parts:

dedicated=gpu:NoSchedule
│         │   │
key       value   effect

• Key — a label-style string (dedicated, node.kubernetes.io/not-ready, nvidia.com/gpu). Same syntax rules as label keys, optional DNS prefix included.
• Value — optional. dedicated=gpu:NoSchedule and spot:NoSchedule are both valid taints; the second has a key and effect but no value.
• Effect — what happens to pods that do not tolerate the taint. This is the part people get wrong, so it gets its own section.

A node can carry multiple taints, and a pod must tolerate all of them to be scheduled there. One untolerated taint is enough to keep the pod off.

The three effects, and how they actually behave

NoSchedule is the workhorse. It affects only future scheduling decisions — every pod already on the node stays put, which is usually what you want when carving out a dedicated pool: taint first, migrate workloads on your own schedule.

PreferNoSchedule is advisory. The scheduler treats it as a preference, not a rule, so under pressure pods land on the node anyway — which makes it nearly useless for isolation. If the point of the taint is "only GPU workloads here," an effect the scheduler may ignore defeats the purpose.

NoExecute is the aggressive one. It rejects new pods and evicts running pods that lack a matching toleration — apply it to a busy node and everything without one is kicked off immediately. This is the effect Kubernetes itself uses for node failures.

kubectl: add, list, remove

Add a taint:

1# key=value:effect
2kubectl taint nodes node-1 dedicated=gpu:NoSchedule
3
4# key with no value
5kubectl taint nodes node-1 spot:NoSchedule
6
7# taint every node matching a label selector
8kubectl taint nodes -l node.kubernetes.io/instance-type=g5.xlarge dedicated=gpu:NoSchedule

List taints — there is no kubectl get taints, which trips everyone up. Use one of these:

1# taints for one node, human-readable
2kubectl describe node node-1 | grep -A3 Taints
3
4# taints across all nodes in one table
5kubectl get nodes -o custom-columns='NODE:.metadata.name,TAINTS:.spec.taints[*].key'
6
7# full detail with values and effects
8kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

Remove a taint — append a - to the taint you added:

# remove a specific key:effect pair
kubectl taint nodes node-1 dedicated=gpu:NoSchedule-

# remove every effect for a key
kubectl taint nodes node-1 dedicated-

The trailing dash is the entire removal syntax. kubectl taint nodes node-1 dedicated=gpu:NoSchedule- removes exactly that taint; omitting the effect removes all taints with that key.

One operational note: taints added with kubectl taint are imperative state on the node object, and a hand-applied taint on an autoscaled node disappears with the node. For managed capacity, set taints declaratively — EKS managed node groups, GKE node pools, and Karpenter NodePools all support it; the Karpenter NodePool generator scaffolds the taints block for you.

Tolerations: the pod side

A taint on its own just empties a node. To let the right pods in, they need a matching toleration:

1apiVersion: v1
2kind: Pod
3metadata:
4  name: gpu-training
5spec:
6  tolerations:
7    - key: "dedicated"
8      operator: "Equal"
9      value: "gpu"
10      effect: "NoSchedule"
11  containers:
12    - name: trainer
13      image: my-trainer:latest

operator: Equal matches key, value, and effect exactly. operator: Exists matches any value for the key:

1tolerations:
2  # tolerate dedicated=<anything>:NoSchedule
3  - key: "dedicated"
4    operator: "Exists"
5    effect: "NoSchedule"
6
7  # tolerate ALL taints — a blanket pass reserved for cluster-critical
8  # agents, and almost never what your app should do
9  - operator: "Exists"

An empty key with Exists tolerates everything; an empty effect matches all effects for the key. Both are big hammers — a blanket Exists toleration on an app deployment quietly opts it back into every node you have carefully fenced off.

tolerationSeconds applies only to NoExecute and answers "how long may this pod stay after the taint appears":

tolerations:
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 120   # evict me 120s after the node goes unreachable

Without tolerationSeconds, a NoExecute toleration means "stay forever." With it, the pod gets a grace window and is then evicted. This is not an exotic option — every pod you run already has it, as we'll see below.

The crucial asymmetry to internalise: a toleration permits, it does not attract. A pod that tolerates the GPU taint may schedule on the GPU node, but the scheduler is equally happy to put it on any untainted node. If the pod must only run on GPU nodes, you pair the toleration with node affinity. Taints keep the wrong pods out; affinity puts the right pods in. Dedicated pools need both.

Taints vs nodeSelector vs node affinity

Three mechanisms, three different questions:

The last row is the real decision criterion. nodeSelector and affinity are opt-in: they constrain only the pods that carry them, so a teammate's deployment with no selector can land on your expensive GPU node and squat there. A taint is opt-out: it protects the node from every pod that has not been explicitly granted access, including pods that do not exist yet and manifests written by people who have never heard of your node pool.

So the rule of thumb:

• Node affinity / nodeSelector — "this pod needs a particular kind of node" (SSD, availability zone, instance family). Checking what a pod's affinity currently is takes one command: kubectl get pod <pod> -o jsonpath='{.spec.affinity}' — and kubectl describe pod shows Node-Selectors directly.
• Taints — "this node must not run general workloads." Protection of the node, not placement of the pod.
• Both together — a truly dedicated pool: the taint keeps everyone else out, the affinity keeps your workload from straying elsewhere.

For a deeper treatment of how these interact with pod priority and preemption, see taints, tolerations and affinity in the scheduling pipeline.

Patterns that actually justify a taint

Dedicated GPU nodes

The canonical case. GPU instances cost 5–10× a general-purpose node; a log shipper scheduled onto one wastes real money and, worse, its memory request can block a training pod from fitting.

kubectl taint nodes -l gpu=true nvidia.com/gpu=present:NoSchedule

1# GPU workload: toleration (get in) + affinity (only here) + resource request
2spec:
3  tolerations:
4    - key: "nvidia.com/gpu"
5      operator: "Exists"
6      effect: "NoSchedule"
7  affinity:
8    nodeAffinity:
9      requiredDuringSchedulingIgnoredDuringExecution:
10        nodeSelectorTerms:
11          - matchExpressions:
12              - key: gpu
13                operator: In
14                values: ["true"]
15  containers:
16    - name: trainer
17      resources:
18        limits:
19          nvidia.com/gpu: 1

On platforms with the ExtendedResourceToleration admission controller, requesting nvidia.com/gpu in limits injects the toleration automatically — but the explicit version costs four lines and never surprises you. The full setup is covered in running GPU workloads on Kubernetes.

Spot / preemptible isolation

Spot nodes can vanish with two minutes' notice. Tainting them ensures only workloads that explicitly declared "I survive interruption" run there:

kubectl taint nodes -l karpenter.sh/capacity-type=spot spot=true:NoSchedule

Stateless, retry-safe workloads add the toleration; databases and anything with a long shutdown never do. With Karpenter this is declarative — the NodePool's spec.template.spec.taints applies the taint to every node it provisions, which is exactly how Karpenter v1's disruption model expects you to fence off interruptible capacity.

Control-plane nodes

You have been using taints all along — every kubeadm cluster ships with:

node-role.kubernetes.io/control-plane:NoSchedule

That taint is why your workloads never land next to etcd. Single-node dev clusters (kind, minikube, k3s) remove it so anything can schedule; production clusters keep it. If pods are running on your control plane, the first thing to check is whether someone removed this taint.

Draining with NoExecute

kubectl drain is the standard way to empty a node, but a NoExecute taint is the blunt alternative when you want pods evicted and the node fenced in one move:

kubectl taint nodes node-1 maintenance=true:NoExecute

Everything without a matching toleration is evicted immediately, and nothing new arrives. The difference from drain: taint-based eviction does not respect PodDisruptionBudgets, so it can take out every replica of a service at once. Use drain for routine maintenance; keep NoExecute for "this node is compromised/broken and pods must leave now."

The taints Kubernetes applies for you

When a node goes unhealthy, the node controller taints it automatically — NoExecute for the failure taints, NoSchedule for the condition taints:

This is where tolerationSeconds stops being theoretical: the admission controller injects tolerations for not-ready and unreachable with tolerationSeconds: 300 into every pod that does not define its own. That is why pods on a dead node take five minutes to be rescheduled — a toleration timeout, not a bug. Stateless services can set a lower value to fail over faster; too low, and a brief network blip stampedes your whole fleet.

kubectl cordon is, under the hood, the unschedulable taint plus a flag — which is why cordoned nodes reject new pods but keep running the old ones, exactly like NoSchedule.

Debugging: pod stuck Pending because of a taint

The symptom is a pod in Pending with a FailedScheduling event:

kubectl describe pod stuck-pod

Events:
  Warning  FailedScheduling  0/5 nodes are available:
    3 node(s) had untolerated taint {dedicated: gpu},
    2 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }.

The scheduler tells you exactly which taints excluded which nodes — read the message before touching anything. From there the checklist is short:

1. List the taints on candidate nodes with the custom-columns command above.
2. Diff against the pod's tolerations: kubectl get pod stuck-pod -o jsonpath='{.spec.tolerations}'. Remember Equal must match the value exactly — dedicated=gpu is not tolerated by a toleration for dedicated=ml.
3. Decide which side is wrong. Either the pod should tolerate the taint (add the toleration) or the taint should not be there (remove it with the trailing-dash syntax). Do not reflexively add operator: Exists blanket tolerations to make the error go away — that disables node protection cluster-wide for this workload.

Taints are only one of several reasons for FailedScheduling — insufficient CPU/memory requests and affinity mismatches produce the same Pending state. The full decision tree is in fixing Pending pods, and the general methodology in the Kubernetes debugging guide.

Frequently Asked Questions

Does tainting a node evict the pods already running on it?

Only with NoExecute. NoSchedule and PreferNoSchedule affect scheduling decisions exclusively — existing pods keep running. This is deliberate: it lets you taint a node first and migrate workloads gracefully.

Why is my pod scheduled on a node it doesn't tolerate?

It almost certainly was scheduled before the taint was applied (with NoSchedule, which does not evict), or it carries a broader toleration than you think — check for operator: Exists with no key. DaemonSet pods also tolerate the built-in node condition taints by default.

How do I taint a node so that only one app can use it?

Taint the node, then give that app both a toleration and node affinity pinned to the node's labels. The toleration alone lets the app in but does not keep it there, and does not keep it only there.

Can I taint a node group declaratively instead of with kubectl?

Yes, and you should for autoscaled capacity: kubelet --register-with-taints at the node level, or the taints field in EKS managed node groups, GKE node pools, and Karpenter NodePool templates. Imperative taints do not survive node replacement.

What's the difference between cordon and taint?

kubectl cordon sets spec.unschedulable, which manifests as the node.kubernetes.io/unschedulable:NoSchedule taint. It is a taint with a nicer CLI — new pods are rejected, running pods stay. kubectl drain is cordon plus eviction that respects PodDisruptionBudgets.

See also

• Kubernetes Taints and Tolerations: Controlling Pod Placement
• Node Affinity, Taints & Tolerations in Production
• Taints, Tolerations, Affinity and Priority — how taints fit into the full scheduling pipeline
• Fix Kubernetes Pending Pods — every cause of FailedScheduling, not just taints
• Karpenter v1 Deep Dive — declarative taints on autoscaled NodePools
• GPU Workloads on Kubernetes — the dedicated-pool pattern end to end
• Kubernetes Debugging Guide — the wider troubleshooting methodology

Official References

• Assigning Pods to Nodes — nodeSelector, affinity and anti-affinity semantics
• Taints and Tolerations — how taints repel pods and how tolerations override them