How Kubernetes Nodes Learn to Say No: Taints and Tolerations

TL;DR

• Taints go on nodes, tolerations go on pods. A tainted node rejects pods by default, and a toleration is what lets a specific pod back in. It is opposite of node selectors and affinity, where the pod was the one doing the choosing.
• Three effects: NoSchedule, PreferNoSchedule, NoExecute.
• NoExecute is the odd one out. It can evict pods that are already running, not just block new ones from landing.
• Tolerating a taint doesn't mean a pod prefers that node. It just means the taint won't stop it from scheduling on the node.

Skip around if you already know the basics:

• The problem this actually solves
• Taints, tolerations, and the three effects
• Where this actually shows up
• What the scheduler's doing under the hood
• Taints and tolerations vs node affinity
• What goes wrong, and why
• Try it yourself

WHY: the problem

Node selectors and affinity, from Part 2, are opt-in. A pod decides which nodes it wants, and the scheduler tries to honor that.

Sometimes you need the opposite. A node that rejects almost everything by default, unless a pod specifically says it's fine with that. Control-plane nodes are the classic example, you don't want regular application pods landing there and competing with the control plane for resources.

That's what taints and tolerations are for. Instead of a pod pulling toward a node, a node pushes pods away, and only pods that explicitly tolerate that push can get scheduled on it.

WHAT: taints, tolerations, and effects

A taint lives on a node. It's a key, a value, and an effect, written as key=value:effect.

A toleration lives on a pod. It lists the key, value, and effect the pod is willing to tolerate. If it matches, the taint doesn't block that pod.

Here's a way to picture it: a taint is bug repellent sprayed on a node. A toleration is a bug that's immune to that specific spray.

The three effects behave differently, and the analogy maps onto all of them:

• NoSchedule - full-strength spray. Immune bugs land fine, everything else won't come near. Bugs that landed before the spray went down aren't touched either way, this only stops new arrivals.
• PreferNoSchedule - a lighter dose. Bugs would rather avoid it, but if every other spot's taken, they'll land there anyway. Same as above, anything that landed earlier just stays put.
• NoExecute - once the spray's applied, only immune bugs can land or stay. Every other bug gets rejected, whether it landed before the spray or was about to land after.

Setting a taint on a node:

kubectl taint nodes node-1 dedicated=gpu:NoSchedule

Tolerating it from a pod:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  containers:
  - name: nginx
    image: nginx

operator can also be Exists instead of Equal, if a pod should tolerate a key regardless of its value:

tolerations:
- key: "dedicated"
  operator: "Exists"
  effect: "NoSchedule"

A node isn't limited to one taint. When there are several taints, a pod has to tolerate every NoSchedule and NoExecute taint on that node to be allowed there, each one gets checked independently, and missing even one will keep the pod out. PreferNoSchedule taints don't count toward that requirement, they only ever affect scoring, never a hard block.

One more piece worth knowing: tolerationSeconds. It only applies to NoExecute, and it changes what the toleration itself means. Instead of tolerating the taint forever, it's saying "I tolerate this taint, but only for N seconds." The countdown starts the moment the taint shows up. Until it runs out, the pod counts as tolerating the taint and stays put. Once it expires, Kubernetes treats the pod as if it no longer tolerates that taint at all, and evicts it, exactly like a pod that never had a matching toleration in the first place.

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu"
  effect: "NoExecute"
  tolerationSeconds: 3600

📖 Docs: Taints and tolerations

WHEN: where this actually shows up

A few real situations where taints and tolerations show up:

• Control-plane nodes, tainted by default so regular workloads stay off them.
• Dedicated GPU or other specialized-hardware nodes, so only the workloads that need them land there.
• Kubernetes itself uses this mechanism internally. When a node goes unreachable or stops responding, the node controller taints it automatically.

HOW: what's actually happening

Taints get checked during the same filtering phase from Part 1, just inverted. Normally filtering asks "can this node run the pod?" Here it's more like "has this node been told to reject the pod?" If yes and if there's no matching toleration on pod, the node gets filtered out, same as any other hard requirement.

PreferNoSchedule doesn't touch filtering at all. It works through scoring, same as preferred node affinity in Part 2, nudging the scheduler away from tainted nodes without ruling them out.

What happens on a node that's already running pods matters here, (I am intentionally repeating these points to make it stick to your memory) and the three effects don't behave the same way:

• Taint a live node with NoSchedule or PreferNoSchedule, and nothing happens to pods already running there. These only affect future scheduling decisions, not the pods sitting there right now.
• Taint it with NoExecute, and any pod without a matching toleration gets evicted, immediately, or after the grace period if tolerationSeconds is set. This is the one real exception to "scheduling is a one-time decision" from Part 1, worth closing that loop here.

The automatic tainting mentioned in previous section works off a fixed set of built-in keys, all under the node.kubernetes.io/ prefix: not-ready, unreachable, memory-pressure, disk-pressure, pid-pressure, network-unavailable, and unschedulable. The first two carry NoExecute, which is what actually triggers eviction the moment a node's condition flips.

That sounds like it should mean every pod in the cluster gets evicted the instant a node blips offline for a second, but Kubernetes doesn't let that happen by default. Every pod automatically gets a built-in toleration for not-ready and unreachable, capped at 300 seconds, whether or not one was written by hand. That's the buffer that keeps a brief network hiccup from turning into a mass eviction.

DIFFERENCE: taints and tolerations vs node affinity

Node affinity and taints solve a similar-sounding problem from opposite directions.

Affinity is a pod pulling toward a node it wants. Taints are a node pushing pods away by default, and a toleration is what lets a specific pod back in.

Here's where that difference actually shows up. Back in Part 2, node affinity let a pod prefer or require a specific node, but that was only the pod's side of the deal. Nothing stopped some other pod, one with no affinity rule at all, from landing on that same node too. Affinity never restricts the node itself.

Taints and tolerations exist for exactly that gap. If the goal is "no pod should schedule here unless it explicitly tolerates this" affinity can't do that on its own, that's what a taint is for.

Tolerating a taint doesn't mean a pod prefers that node either, or that it'll get scheduled there. It just means the taint won't block it. If the goal is a pod landing on GPU nodes specifically, not just being allowed there, tolerations usually get paired with node affinity on top. The taint keeps everyone else out, the affinity pulls the right pod in.

FAILURE: what goes wrong

The tolerate-doesn't-mean-prefer mix-up is the most common one. A pod with the right toleration can still land on an untainted node it has nothing to do with, because a toleration only removes a restriction, it doesn't express a preference.

The second one is more of a hazard than a misconception. Adding a NoExecute taint to a node that's already serving traffic can evict running pods with no warning, if that wasn't accounted for. Worth checking what's already running and what it tolerates before tainting a node people are relying on.

One more thing worth flagging now and covering properly later: DaemonSet pods tolerate several of these built-in taints automatically, which is part of why they keep running in places regular pods can't. More on that in Part 5.

PRACTICE: taint, tolerate, check

Taint a node:

kubectl taint nodes node-1 dedicated=gpu:NoSchedule

Deploy the toleration pod from the WHAT section, plus a plain one with no toleration at all:

apiVersion: v1
kind: Pod
metadata:
  name: plain-pod
spec:
  containers:
  - name: nginx
    image: nginx

kubectl apply -f gpu-pod.yaml
kubectl apply -f plain-pod.yaml
kubectl get pods -o wide

plain-pod should avoid node-1 entirely. Confirm the taint is actually there:

kubectl describe node node-1

Look for a Taints: line near the top of the output.

To remove the taint later:

kubectl taint nodes node-1 dedicated=gpu:NoSchedule-

That trailing - is easy to miss and just as easy to forget, so it's worth calling out on its own.

Next up, Part 4: pod affinity and anti-affinity, where instead of a pod caring about a node, it starts caring about other pods.

If anything here didn't land, or you'd explain it differently, drop a comment. Doubts and pushback are both welcome.