The scheduler
First rule nodes out, then pick the best of the rest
Filter out nodes that cannot fit the pod. Score the rest.
Every new pod needs a node, and the scheduler decides which, in two passes. This pod requests one C P U. The grey bars show how much of each node other pods have already requested. First, filtering. Node A has only half a core left unrequested, so it is ruled out. Then scoring. Nodes B and C could both take it. B has the most room, so B scores highest. The pod is bound to node B, and its request is added to the bar.
Placement
Five pods, three nodes, and the reason for each
Each placement follows from a label, a taint or a rule.
Room is only one consideration. Watch five pods being placed on three slightly different nodes. The first web pod has no special needs, and lands on A. The second carries a rule to stay apart from other web pods, so it goes to B. The database asks for a node labelled with fast disks. Only A has that label. The training job tolerates the taint on the G P U node and asks to go there. And the cache has no toleration, so node C is closed to it. It goes to B, which has more room.
Affinity, written down · Going deeper
The rules behind that placement, as written in the pod
required filters nodes. preferred only scores them.
Steering
Affinity draws pods in; a taint pushes them away
Affinity: the pod chooses. Taint: the node refuses. Anti-affinity: pods keep apart.
Those reasons come from three tools. Node affinity, and its simpler form the node selector, is written on the pod. The pod says which nodes it wants, by label. A taint is written on the node. The node turns away every pod that does not carry a matching toleration. And anti affinity, or a spread rule, is about pods and other pods: keep my copies on different nodes or zones, so one failure cannot take them all.
Steering · Going deeper · 1 of 2
Lab: why is my pod Pending?
Taint the node, watch a pod wait, and read the reason.
Taint effects · Going deeper · 2 of 2
Three taint effects, and the toleration as written
NoSchedule stops new pods. NoExecute also evicts running ones.
Scaling pods
The autoscaler holds load at a target by changing the count
pods × load ÷ target, rounded up. That is the new count.
Placement decides where pods run. Autoscaling decides how many. You set a target, here fifty percent C P U, and the autoscaler does one sum, again and again. In the morning two pods run at forty percent. The sum gives one point six, which rounds up to two. No change. At noon the load jumps to ninety. Two times ninety over fifty is three point six. Scale to four. With four pods the average falls to forty five, close to the target. In the evening it drops to twenty, and the sum says two. Two pods, at forty percent again.
The HPA, written down · Going deeper
The autoscaler as written, and the three defaults behind it
Every 15 seconds, a 10% tolerance, and 5 minutes before scaling down.
Scaling nodes
When pods no longer fit, the cluster itself grows
Pending pods make the cluster add a node.
More pods need somewhere to run. Suppose each node holds four of these pods, both nodes are full, and the autoscaler asks for ten. Eight are running. Two are stuck in the state called pending, because the scheduler finds no node with room. A cluster autoscaler notices pods pending for that reason, and asks the cloud for one more machine. When the node joins, the scheduler places the two waiting pods on it.
Lesson complete
You finished “Scheduling and scaling”
Nicely done.