The problem

Package once, and let the cluster keep it running

# DockerfileFROM python:3.12-slimCOPY . /appRUN pip install -r ...CMD ["gunicorn", "api:app"]build, pushImageapi:1.0 in a registrydeployKubernetes cluster3 replicas wanted, behind one addressrolling updates, no downtimerestarts anything that diesPackage once, run anywhere, and let the cluster keep it running. Then see what happens when it cannot.A small team has a Python web API. They package it as an image, push it to a registry and run 3 copies on Kubernetes behind oneaddress, so a copy can die without users noticing and new versions roll out without downtime. Version 1 works. Version 2 adds onefeature, a cache warmed at start-up, and the pods start restarting over and over. Follow it from the Dockerfile to the fix.

A Python API as an image, 3 replicas on Kubernetes. Version 2 adds a cache warmed at start-up.

The first deployment

A Deployment keeps pods running; a Service gives them one address

Ingressapi.example.comServiceone stable IPDeployment: 3 replicasReplicaSet api:1.0podpodpodImage size, to scale1,100 MB, one stage180 MB, multi-stage buildA Deployment keeps the pods running, a Service gives them one address, and an Ingress brings in the web.The Deployment asks for 3 replicas; its ReplicaSet creates the pods and replaces any that die. A Service gives them one stable address andonly sends traffic to pods that are ready. An Ingress maps the domain to the Service. A multi-stage build leaves the compilers behind:the image falls from 1,100 MB to 180 MB, so every node pulls it faster. Version 1 runs 3 of 3 ready.

Deployment, ReplicaSet, 3 pods, a Service and an Ingress. A multi-stage build cuts the image to 180 MB.

Version 2

The rollout stalls safely while version 2 restarts again and again

Rolling update, default settingsv1v1v1v2max surge 1 extra pod, max unavailable 0: v1 stays until v2 is ready$ kubectl get podsNAME READY STATUS RESTARTS AGEapi-7d9c6b5f4-k2x8q 0/1 CrashLoopBackOff 5 (90s ago) 6mapi-5b8f7c9d6-9hq2m 1/1 Running 0 2dThe rollout stalls safely: users still reach version 1, while version 2 restarts again and again.With the defaults for 3 replicas, the Deployment may add 1 extra pod and take none away, so it starts one v2 pod and waits for it to be ready.It never is. The container starts, dies and is restarted, 5 times in six minutes, with the waits growing each time: 10, 20, 40, 80, 160 seconds.That is CrashLoopBackOff. The rolling update did its job: no user is affected. The question is why the pod dies.

One v2 pod starts and never becomes ready: CrashLoopBackOff, 5 restarts. Users still reach v1.

The cause

Killed for using more memory than its limit

$ kubectl describe pod api-7d9c6b5f4-k2x8q Last State: Terminated Reason: OOMKilled Exit Code: 137 Limits: memory: 256Mi$ kubectl logs api-7d9c6b5f4-k2x8q --previousINFO starting gunicorn, 2 workersINFO warming cache: loading fares ...(nothing more: the process was killed)Memory at start-up, to scalesteady 300 Mipeak 410 Milimit 256 MiThe container is killed for using more memory than its limit: 410 Mi at start-up against 256 Mi allowed.describe shows the last exit: OOMKilled, exit code 137, which is 128 plus signal 9, a kill the process cannot catch. logs --previous shows theoutput of the container that died: it stopped in the middle of warming the new cache. Measured with the same image on a laptop, start-uppeaks at 410 Mi before settling to 300 Mi. Version 1 never needed more than about 200 Mi, so the old limit was never tested.

describe: OOMKilled, exit code 137, limit 256 Mi. Start-up peaks at 410 Mi.

The fix

Size the limit from the measured peak, and give a slow start its own probe

Beforeresources: requests: { memory: 128Mi } limits: { memory: 256Mi }livenessProbe: httpGet: { path: /healthz, port: 8000 }Afterresources: requests: { memory: 448Mi } limits: { memory: 512Mi }startupProbe: httpGet: { path: /healthz, port: 8000 } periodSeconds: 2, failureThreshold: 30livenessProbe: (unchanged)3 of 3 ready on v2; rollout completeSize the limit from the measured peak, and give a slow start its own probe.The limit becomes 512 Mi, the measured peak of 410 Mi with about 25% headroom, and the request 448 Mi, so the scheduler places the podonly where that memory exists. Warming takes about 25 seconds, so a startup probe allows up to 60 (30 checks, 2 s apart) before theliveness probe takes over and can restart the pod. The rollout continues: 3 of 3 pods on v2, and no user ever saw an error.

Request 448 Mi, limit 512 Mi, and a startup probe that allows 60 seconds.

The trade-off

Every megabyte requested is one no other pod can use

Pods that fit on one node with 3,584 Mi for pods, by memory requestRequest 128 Mi28 podsRequest 448 Mi8 podsToo lowpods packed onto nodes that cannot hold their real use: evicted or killed when memory runs shortToo highmemory reserved and never used: more nodes, a bigger billChosenrequest near steady use (448 Mi), limit above the measured peak (512 Mi); review after a week of real dataRequests buy a guaranteed place on a node; every megabyte reserved is one no other pod can use.The scheduler packs pods by their requests, not their use. At 128 Mi, 28 of these pods fit on a node and each can still grow to 410 Mi, which thenode cannot hold for all of them. At 448 Mi only 8 fit, so the same work needs about twice the nodes. Set requests from measured steadyuse, limits from the measured peak, and check both against real metrics once the version has run for a while.

At 128 Mi, 28 pods fit on a node; at 448 Mi, 8. Requests near steady use, limits above the peak.

The finished picture

A healthy workload is six small, measured decisions

1 Small imagemulti-stage, 180 MBLesson 3: images2 Deployment3 replicas, rolling updateLesson 5: workloads3 Service and Ingressonly ready pods get trafficLesson 6: networking4 Requests and limits448 Mi and 512 Mi, measuredLesson 7: configuration5 Probesstartup, then livenessLesson 12: operating6 Debugging pathget, describe, logs --previousLesson 12: operatingEach part opens the lesson that teaches it. The two in colour are what this case changed.A healthy workload is six small decisions, each one checked against something measured.Nothing here is exotic. The image is small, the Deployment rolls out safely, the Service hides pods that are not ready, the resourcesmatch measured use, the probes match how the app really starts, and the team knows the three commands that explain a crash.Version 2 failed safely because the first three were already right.

Small image, Deployment, Service and Ingress, measured resources, probes and the debugging path. Each links to its lesson.

A real case

At Monzo in 2017, a stuck rollout became a full outage

Etcd upgraded3 nodes to 9,two weeks earlierA deploy at 14:10payments fail;rollback does not helpMesh restartedlinkerd cannot readan empty ServiceWhole platform downfrom 15:13Services deletedat 15:31,it recoversChangedfix the Kubernetes timeout bug; a linkerd version that parses empty Services; better health checks,dashboards and alerts; clearer ways to tell staff and customers during an outageAt Monzo in October 2017, a stuck rollout became a full outage when the fix made it worse.Monzo’s head of engineering wrote that a deployment broke payments and rolling back did not help, because the service mesh had stoppedhearing where pods were. Restarting the mesh, a decision the write-up calls poor, took the whole bank down for about 1½ hours of theafternoon. Like our case: read what the system says before changing it, and know which part of the platform decides where traffic goes.

A deploy broke payments; restarting the service mesh took the whole bank down for about an hour and a half.

Lesson complete

You finished “The pod that keeps restarting”

Nicely done.

Open in a new tab