Finding the fault
The status tells you which question to ask next
Pending, ImagePullBackOff, CrashLoopBackOff: each points to one command.
Something is broken. Where do you start? With the status column of kubectl get pods, because each status points somewhere different. Pending means the pod has no node. The scheduler's reason is in the pod's events. Image pull back off means a node was chosen but cannot fetch the image. The event tells you which image. Crash loop back off means the image runs, and the program keeps exiting. Its own logs say why, from the run before. And sometimes the pod is running and nobody can reach it. Then ask whether the Service lists the pod at all.
Finding the fault · Going deeper
Lab: break two pods and find out why
A wrong tag and a crash: read the cause from the cluster.
Evidence
Work outwards: container, pod, node, cluster
logs, then describe, then the node, then events.
There are four places to look, and a sensible order. The container: its logs are simply what the program printed. Add previous to read the run that crashed. The pod: describe shows what Kubernetes knows about it. How many restarts, the last exit code, and its recent events. The node: is it short of memory or disk? What else is running there? And the cluster as a whole: the list of events, in order, shows what happened and when.
Probes
Probes tell Kubernetes what healthy means
Readiness failing stops traffic. Liveness failing restarts.
Kubernetes can only heal what it can see. By default it sees one thing: whether the process exists. Probes are checks you define, and there are three. A startup probe asks whether the program has finished starting, and holds the other two back until it has. A readiness probe asks whether it can serve right now. If not, the pod is taken out of the Service. It is not restarted. A liveness probe asks whether it is stuck for good. If so, the container is restarted. In a pod's first minute, that means no traffic arrives until it reports ready.
Probe timings · Going deeper
Probes as written: period times failures is how long Kubernetes waits
Startup: 60 seconds allowed. Liveness: restart after 30 seconds of failures.
Upgrades
Control plane first, then one node at a time
The control plane leads. Each node is drained, upgraded, returned.
Kubernetes releases a new version several times a year, and each is supported for about a year, so upgrading is routine work. It follows a fixed order. Before, everything runs one point three three. The control plane goes first, because the nodes may lag behind it but must never be ahead. Then one node. It is drained, which moves its pods to other nodes. Its software is upgraded, and it is put back into service. Then the next node, and so on.
kubeadm upgrade · Going deeper
The kubeadm upgrade, command by command
kubeadm first, then drain, then the kubelet, then uncordon.
Backup
etcd is the cluster's memory: back it up
A restore returns the cluster to the last snapshot.
One component cannot be rebuilt from anything else: et cee dee, which records every object in the cluster. So it is backed up by taking snapshots. Here a snapshot is saved at two in the morning. A new Deployment is created at nine. And at two in the afternoon, et cee dee is lost. You restore the snapshot. The cluster now remembers the world as it was at two in the morning. Twelve hours are missing. The Deployment from nine o'clock was never in that snapshot, so the cluster has forgotten it. But data inside volumes was never stored in et cee dee, so it is untouched.
Snapshot and restore · Going deeper
The snapshot and the restore, as typed on a control plane node
snapshot save with three certificates. snapshot restore into a new directory.
Lesson complete
You finished “Operating and fixing a cluster”
Nicely done.