The problem
Package once, and let the cluster keep it running
A Python API as an image, 3 replicas on Kubernetes. Version 2 adds a cache warmed at start-up.
A small team writes a Dockerfile for its Python API. It builds an image and pushes it to a registry. Kubernetes runs 3 copies behind one address, rolls out new versions without downtime, and restarts anything that dies. Version one works. Version two adds a cache, warmed at start-up, and its pods begin restarting over and over.
The first deployment
A Deployment keeps pods running; a Service gives them one address
Deployment, ReplicaSet, 3 pods, a Service and an Ingress. A multi-stage build cuts the image to 180 MB.
The Deployment asks for 3 replicas, and its ReplicaSet creates the pods and replaces any that die. A Service gives them one stable address and sends traffic only to pods that are ready, and an Ingress brings the domain to it. A multi-stage build leaves the compilers behind, so the image shrinks from over a gigabyte to 180 megabytes.
Version 2
The rollout stalls safely while version 2 restarts again and again
One v2 pod starts and never becomes ready: CrashLoopBackOff, 5 restarts. Users still reach v1.
With the default settings, the Deployment adds one version two pod and keeps all three old ones until the new one is ready. It never is. It starts, dies, and is restarted, 5 times in six minutes, waiting longer each time. That is CrashLoopBackOff. And no user has noticed, because version one is still serving.
The cause
Killed for using more memory than its limit
describe: OOMKilled, exit code 137, limit 256 Mi. Start-up peaks at 410 Mi.
kubectl describe shows how the last container ended: OOMKilled, exit code one three seven. That is a kill signal the process cannot catch. The memory limit is 256 mebibytes. logs with the previous flag shows what the dead container printed: it stopped in the middle of warming the cache. Run on a laptop, start-up peaks at 410 mebibytes before settling at 300. The old limit was never tested by version one.
The fix
Size the limit from the measured peak, and give a slow start its own probe
Request 448 Mi, limit 512 Mi, and a startup probe that allows 60 seconds.
Before: a small request, a 256 mebibyte limit, and only a liveness probe. After: the limit is 512, the measured peak with some headroom, and the request is 448, so the pod lands only where that memory exists. A startup probe gives the slow start up to a minute before the liveness probe can restart it. The rollout finishes: all 3 pods on version two, and no user saw an error.
The trade-off
Every megabyte requested is one no other pod can use
At 128 Mi, 28 pods fit on a node; at 448 Mi, 8. Requests near steady use, limits above the peak.
The scheduler packs pods by their requests. At a hundred and twenty-eight mebibytes, 28 of these pods fit on a node. At 448, only 8. Too low, and pods are packed onto nodes that cannot hold their real use. Too high, and you pay for memory nobody uses. Set requests from steady use, limits from the peak, and check them against real data.
The finished picture
A healthy workload is six small, measured decisions
Small image, Deployment, Service and Ingress, measured resources, probes and the debugging path. Each links to its lesson.
Here is the finished workload. A small image. A Deployment that rolls out safely. A Service that hides pods that are not ready. Requests and limits from measured use. Probes that match how the app really starts. And three commands that explain a crash: get, describe, and logs previous. Each part opens the lesson that teaches it.
A real case
At Monzo in 2017, a stuck rollout became a full outage
A deploy broke payments; restarting the service mesh took the whole bank down for about an hour and a half.
Two weeks before, Monzo had grown its etcd cluster from three nodes to nine. On the twenty-seventh of October 2017, a deployment broke payments, and rolling back did not help: the service mesh, linkerd, had stopped hearing where pods were running. The team restarted linkerd. It could not parse a Kubernetes Service with no pods behind it, and the whole platform went down. Deleting those empty Services brought it back. Monzo fixed the bugs and added better health checks and alerts. Like our case: read what the system tells you before you change it.
Lesson complete
You finished “The pod that keeps restarting”
Nicely done.