The scheduler

First rule nodes out, then pick the best of the rest

New podrequests 1 CPUCPU already requested, of 4 coresNode A3.5 requested, 0.5 freeNode B1 requested, 3 freeNode C2 requested, 2 free✗ too little free✓ most roomThe scheduler first rules nodes out, then picks the best of those left.Filtering removes every node that cannot run the pod: too little unrequested CPU or memory, ataint, a label that is missing. Scoring ranks the rest. The scheduler counts what pods haverequested, never what they are using, so a pod without requests looks as if it needs nothing.

Filter out nodes that cannot fit the pod. Score the rest.

Placement

Five pods, three nodes, and the reason for each

Node Azone a, label disk=ssdNode Bzone bNode Czone a, taint gpu:NoScheduleweb-1web-1to Node A:any node will doweb-2web-2to Node B:spread rule: not beside another web poddb-0db-0to Node A:nodeSelector disk=ssd, and only A has that labeltrain-1train-1to Node C:it tolerates the gpu taint and asks for a gpu nodecache-1cache-1to Node B:no toleration, so C is closed; B has more room than AEvery placement has a reason, and the reason is written in the pod or on the node.Labels and taints on nodes, with selectors, tolerations and spread rules on pods, are how yousteer the scheduler. Give it no rules and it simply balances the load.

Each placement follows from a label, a taint or a rule.

Affinity, written down · Going deeper

The rules behind that placement, as written in the pod

db-0: only on nodes labelled disk=ssdweb: spread evenly across the nodesspec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: disk operator: In values: [ssd]spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: { app: web }The short form of the same rule:nodeSelector: { disk: ssd }maxSkew 1: node counts of web pods differ by 1 at mosttopologyKey: what counts as one place (a node, or a zone)ScheduleAnyway would make the rule a preference"required" is a filter: with no matching node the pod stays Pending. "preferred" only adds to a node's score.The long name has two halves. DuringScheduling: checked when the pod is placed. IgnoredDuringExecution: if the label islater taken off the node, the pod is not moved. Operators are In, NotIn, Exists, DoesNotExist, Gt and Lt. A preferred rulecarries a weight from 1 to 100. podAntiAffinity is written the same way, with a topologyKey to say what "apart" means.

required filters nodes. preferred only scores them.

Steering

Affinity draws pods in; a taint pushes them away

Node affinityor the simpler nodeSelectorpodnodeThe pod chooses nodes:"only nodes labelled disk=ssd"Taint and tolerationset on the nodepodnodeThe node refuses pods:"keep out, unless you tolerate gpu"Pod anti-affinityor a topology spread rulepodpodPods avoid each other:"not on a node that has my twin"Affinity draws pods towards nodes. A taint pushes them away.They are not opposites that cancel out. A toleration only permits a pod to use a tainted node; itdoes not send it there. To reserve nodes for one team, use both: a taint to keep others out andaffinity to bring that team's pods in. Control plane nodes are tainted, which keeps your pods off them.

Affinity: the pod chooses. Taint: the node refuses. Anti-affinity: pods keep apart.

Steering · Going deeper · 1 of 2

Lab: why is my pod Pending?

TypeLook forkubectl get nodeskubectl taint nodes kind-control-plane demo=yes:NoSchedulekubectl run p1 --image=nginxkubectl get pod p1kubectl describe pod p1 | tail -3kubectl taint nodes kind-control-plane demo-the node name, here kind-control-planenode/kind-control-plane taintedpod/p1 createdSTATUS Pending, and it stays so0/1 nodes are available: untolerated taintuntainted; p1 is Running moments laterLab: close a node with a taint and read why a pod is waiting.On a one-node cluster the taint closes the only node, so the pod has nowhere to go.The Events at the end of "kubectl describe pod" are where the scheduler explains itself.

Taint the node, watch a pod wait, and read the reason.

Taint effects · Going deeper · 2 of 2

Three taint effects, and the toleration as written

The taint, put on the node; the toleration, written in the pod spec$ kubectl taint nodes node-c gpu=true:NoScheduletolerations:- key: gpu operator: Equal value: "true" effect: NoSchedulekey=value:effect. Remove it with a final minus: gpu-or Exists, with no value: tolerate any gpu taintleave effect out to tolerate every effectEffectA pod without a matching tolerationA pod already running on the nodeNoScheduleis never placed herestaysPreferNoScheduleis placed here only if nowhere else fitsstaysNoExecuteis never placed hereis evicted, at once or after tolerationSecondsNoSchedule closes the door to new pods. NoExecute also empties the room.Kubernetes uses NoExecute itself: a node that stops answering is tainted node.kubernetes.io/unreachable, and every pod isgiven a toleration for it that lasts 300 seconds. That is why pods on a dead node are replaced after about five minutes,not at once. Control plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule.

NoSchedule stops new pods. NoExecute also evicts running ones.

Scaling pods

The autoscaler holds load at a target by changing the count

TimePodsAverage CPUtarget 50%pods x CPU / targetDecision09:0040%2 x 40 / 50 = 1.6stay at 212:0090%2 x 90 / 50 = 3.6scale to 412:0545%4 x 45 / 50 = 3.6stay at 420:0020%4 x 20 / 50 = 1.6scale to 220:1040%2 x 40 / 50 = 1.6stay at 2The autoscaler keeps the average load near a target by changing the number of pods.Every few seconds it works out one formula, rounds the answer up, and sets the replica count.CPU is measured as a share of each pod's request, so pods without requests cannot be scaledthis way. It adds pods quickly and removes them only after the load has stayed low for five minutes.

pods × load ÷ target, rounded up. That is the new count.

The HPA, written down · Going deeper

The autoscaler as written, and the three defaults behind it

apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: name: webspec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: web minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50The defaults behind itchecks the metrics every 15 secondsdoes nothing while load is within 10% of the targetremoves pods only after 5 minutes of lower load; adds them at oncethe workload whose replica count it setsnever fewer than thisnever more, whatever the loadaverage CPU in use, as a share of each pod's requestthe 50% target of the tableOne command makes the same object: kubectl autoscale deployment web --cpu-percent=50 --min=2 --max=10.It needs metrics-server in the cluster and a CPU request on the pods. Without either, kubectl get hpa shows <unknown> underTARGETS and nothing is scaled.

Every 15 seconds, a 10% tolerance, and 5 minutes before scaling down.

Scaling nodes

When pods no longer fit, the cluster itself grows

Each node has room for 4 of these pods. The autoscaler now asks for 10.Node 1: fullNode 2: full2 pods Pendingno node has roomNode 3: addeda new machine, minutes laterPods scale first. When they no longer fit, the cluster itself grows.A cluster autoscaler watches for pods that are Pending because no node can hold them, and addsa node from the cloud. It also removes nodes that have stayed mostly empty, moving their podsfirst. Pods appear in seconds and machines take minutes, so keep a little spare room.

Pending pods make the cluster add a node.

Lesson complete

You finished “Scheduling and scaling”

Nicely done.

Open in a new tab