Finding the fault

The status tells you which question to ask next

kubectl get podsread the STATUSPendingno node will take the pod: too little room, a taint, or a claim that is not boundkubectl describe pod web-x2k4qread the Events at the endImagePullBackOffthe node cannot fetch the image: a wrong name or tag, or a private registrykubectl describe pod web-x2k4qthe event names the imageCrashLoopBackOffthe container starts, exits, and is restarted, with a longer wait each timekubectl logs web-x2k4q --previousits output before it diedRunning, no answersthe pod is healthy but requests do not reach it: a label, a selector or a probekubectl get endpointslicesis the pod behind the Service?The status tells you which question to ask next. Then one command answers it.Most broken pods show one of a handful of states, and each state points to one place to look.Work from the pod outwards: its status, its events, its logs, and only then the node and the cluster.

Pending, ImagePullBackOff, CrashLoopBackOff: each points to one command.

Finding the fault · Going deeper

Lab: break two pods and find out why

TypeLook forkubectl run bad --image=nginx:nopekubectl get pod badkubectl describe pod bad | tail -5kubectl set image pod/bad bad=nginxkubectl run crash --image=busybox -- sh -c 'echo boom; exit 1'kubectl logs crash --previouspod/bad createdErrImagePull, then ImagePullBackOffFailed to pull image "nginx:nope"image updated; the pod is Running soon afterpod/crash createdboom: the output of the run that failedLab: break two pods on purpose, and find out why from the cluster."kubectl get pod crash" shows CrashLoopBackOff and a rising count under RESTARTS.Clean up with "kubectl delete pod bad crash".

A wrong tag and a crash: read the cause from the cluster.

Evidence

Work outwards: container, pod, node, cluster

Look atWithIt showsThe containerkubectl logs web-x2k4q --previousthe program's own outputThe podkubectl describe pod web-x2k4qstate, restarts, exit code, eventsThe nodekubectl describe node node-bpressure, capacity, its podsThe clusterkubectl get events --sort-by=.lastTimestampall that happened latelyStart at the container and work outwards: logs, pod, node, cluster.A container's logs are whatever it wrote to standard output. "describe" adds what Kubernetesknows: the last exit code (137 means killed for memory) and recent events. Events are kept forabout an hour only, so look soon. If a whole node is NotReady, its kubelet log is the place.

logs, then describe, then the node, then events.

Probes

Probes tell Kubernetes what healthy means

ProbeThe questionIf the answer is noStartuphas the program finished starting?keep waiting: the other probes are held backReadinesscan it take requests right now?taken out of the Service; not restartedLivenessis it stuck beyond recovery?the container is restartedA pod's first minute0 s: starting, no traffic yet20 s: ready, and added to the ServiceProbes tell Kubernetes what "healthy" means for your program.Without probes, a pod counts as healthy the moment its process exists. Readiness keeps trafficaway from a pod that cannot serve yet, and it is what a rolling update waits for. Liveness is thelast resort for a program that has hung. Pointing liveness at a slow dependency causes restart storms.

Readiness failing stops traffic. Liveness failing restarts.

Probe timings · Going deeper

Probes as written: period times failures is how long Kubernetes waits

startupProbe: httpGet: { path: /healthz, port: 8080 } periodSeconds: 5 failureThreshold: 12readinessProbe: httpGet: { path: /ready, port: 8080 } periodSeconds: 5livenessProbe: httpGet: { path: /healthz, port: 8080 } periodSeconds: 10 failureThreshold: 312 x 5 = 60 seconds to start, or the container is restartedheld back, like liveness, until the startup probe has passedthen checked every 5 seconds for the whole life of the pod3 x 10 = 30 seconds of failures, then a restartFieldinitialDelaySecondsperiodSecondstimeoutSecondssuccessThresholdfailureThresholdDefault010113A probe is a check, a period and a count of failures. Multiply the last two to see how long Kubernetes waits.The default timeout is 1 second: a health endpoint that sometimes takes longer fails its liveness probe, and a healthy pod isrestarted. Checks come in four kinds: httpGet (any status from 200 to 399 passes), tcpSocket (the port opens), exec (thecommand exits with 0) and grpc. Give a slow starter a startup probe; do not stretch the liveness probe to cover the start.

Startup: 60 seconds allowed. Liveness: restart after 30 seconds of failures.

Upgrades

Control plane first, then one node at a time

Control planeNode 1Node 2Beforev1.33v1.33v1.33Step 1v1.34v1.33v1.33control plane firstStep 2v1.34v1.34v1.33drain, upgrade, uncordonStep 3v1.34v1.34v1.34then the next nodeUpgrade the control plane first, then the nodes, one at a time.A node may run an older version than the API server, never a newer one, and you move one minorversion at a time. Draining a node moves its pods elsewhere before the work starts, so anapplication with several replicas keeps serving throughout.

The control plane leads. Each node is drained, upgraded, returned.

kubeadm upgrade · Going deeper

The kubeadm upgrade, command by command

TypeLook forapt-get install -y kubeadm='1.34.1-*'kubeadm upgrade plankubeadm upgrade apply v1.34.1kubectl drain cp1 --ignore-daemonsetsapt-get install -y kubelet='1.34.1-*' kubectl='1.34.1-*'systemctl daemon-reload && systemctl restart kubeletkubectl uncordon cp1kubectl get nodesthe new kubeadm only: nothing is upgraded yetthe versions you can move to, and what will changeSUCCESS! Your cluster was upgraded to "v1.34.1"node/cp1 cordoned, then its pods evictedthe node's own programsno outputnode/cp1 uncordonedcp1 at v1.34.1; the workers still at v1.33The upgrade of the first control plane node, v1.33 to v1.34, as root on a Debian or Ubuntu node named cp1.Before step 1, point apt at the v1.34 package repository (pkgs.k8s.io keeps one per minor version) and apt-mark unhold thepackages. A worker is shorter: install kubeadm, run kubeadm upgrade node in place of plan and apply, then steps 4 to 7.A kubelet may be up to three minor versions older than the API server, never newer. kubeadm moves one minor version at a time.

kubeadm first, then drain, then the kubelet, then uncordon.

Backup

etcd is the cluster's memory: back it up

02:00 snapshot saved09:00 api is created14:00 etcd is lostetcdRestored12 hours of changes are not in the snapshot✗the Deployment api, made at 09:00: unknown to the restored cluster. Apply it again.✓data inside volumes: untouched, because it never lived in etcdetcd is the cluster's memory. A restore takes you back to the last snapshot.Every object you have ever applied is stored in etcd and nowhere else. Take snapshots on aschedule and keep them off the cluster. Manifests kept in Git cover the gap: apply them, or letGitOps apply them, and the lost hours return.

A restore returns the cluster to the last snapshot.

Snapshot and restore · Going deeper

The snapshot and the restore, as typed on a control plane node

On a kubeadm control plane node, as root$ ETCDCTL_API=3 etcdctl snapshot save /backup/etcd.db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key$ etcdutl snapshot status /backup/etcd.db --write-out=table$ etcdutl snapshot restore /backup/etcd.db --data-dir=/var/lib/etcd-restored$ vi /etc/kubernetes/manifests/etcd.yamlask the running etcd for a snapshotetcd insists on a client certificate:these three files are on everykubeadm control plane nodecheck it: hash, revision, keys, sizeunpack into a new, empty directorypoint the etcd-data hostPath at itetcd members1357Needed to agree (quorum)1234May be lost0123A restore does not touch the running etcd. It builds a new data directory, and you point etcd at it.etcd runs as a static pod, so saving the edited manifest is the restart: the kubelet sees the file change and starts etcd onthe restored data. Give it a minute, then check with kubectl get pods -A. Restore with etcdutl: etcdctl snapshot restore hasbeen deprecated since etcd 3.5. With 3 members a cluster survives the loss of 1; an even number adds cost and no safety.

snapshot save with three certificates. snapshot restore into a new directory.

Lesson complete

You finished “Operating and fixing a cluster”

Nicely done.

Open in a new tab