Size for the busiest hour, because the average hides it
Capacity planning provisions for the peak, the spike on top of it, and growth. The average is for the spreadsheet.
Shipping a service is the first ten minutes of its life. Operations is the next ten years, and it starts with a simple question: how much hardware? Here is a day of traffic. Quiet overnight, then a climb. The dashed line is the daily average, the number most dashboards report. But the evening peak is four times that average. For a typical web service the peak is three to five times the mean; for an audience in a single time zone, eight to ten. And on top of the peak sits the spike nobody forecast. So capacity is sized for the peak, plus the spike, plus growth. Eight percent a month doubles traffic in nine months, which is why capacity is reviewed every quarter.
Capacity planning · Going deeper
The load profile
Three numbers describe load: the average, the peak, and their ratio. Compute monthly growth from six months of traffic and project it forward. Under-provisioning costs the service-level objective; over-provisioning costs the cloud bill. The right margin depends on how spiky the traffic is and how fast new capacity can start.
Utilisation and latency
Latency stays flat, then explodes: keep servers under 70% busy
Queueing makes response time rise hyperbolically with utilisation. About 70% at peak is the practical ceiling.
Why not simply run every server at ninety-five percent and save money? Because of this curve. Across the bottom, how busy the server is. Up the side, how long a request takes, as a multiple of the work itself. At fifty percent busy, a request takes twice its own work time, because sometimes it queues behind another. At seventy percent, a little over three times. At ninety percent, ten times. Past about seventy percent the curve bends sharply upward. That bend is the knee. The formula is one line: time equals one over capacity minus arrivals. As arrivals approach capacity, the wait grows without limit. Running hot saves on servers and spends your latency.
Utilisation and latency · Going deeper
The M/M/1 curve
For a single server with random arrivals, response time is W = 1 / (μ − λ), which goes to infinity as utilisation approaches one. Real fleets with many servers have a gentler curve of the same shape. Past the line where arrivals meet service rate there is no graceful degradation: work piles up until something breaks.
Little’s law
Work in flight = arrival rate × time each request spends inside
Little’s law: L = λ × W. A dependency that slows down fills your worker pool without one extra request arriving.
One multiplication explains most queueing surprises. Five hundred requests arrive each second. Each spends sixty milliseconds inside. Multiply, and thirty requests are in flight at any instant. With a pool of thirty-two workers, that just fits: thirty busy, two free. Now a dependency gets slower, and each request takes eighty milliseconds. Traffic has not changed. But five hundred times zero point zero eight is forty. Forty requests need a worker, there are thirty-two, and eight must wait. Waiting makes them slower, which puts more in flight. This is how one slow dependency spreads.
Little’s law · Going deeper
Little’s law
L = λW holds for any stable queueing system regardless of its internal design: web servers, message queues, thread pools, garbage collectors. Knowing any two quantities fixes the third. An autoscaling threshold is in effect a target W for an expected λ.
Check yourself
How many requests are inside this service at any instant?
Arrival rate × time inside: 200 × 0.1 = 20. If each request slowed to 0.2 s, 40 would be in flight with no extra traffic.
Autoscaling
Autoscaling always arrives late, and users pay for the gap
Autoscaling follows load with a delay. Headroom, prediction and warm pools are what cover the delay.
Fixed capacity is wrong twice a day: too much at four in the morning, too little at six in the evening. So a controller adds and removes machines. Watch what happens when load doubles in sixty seconds. The controller notices and asks for machines, but each takes ninety seconds to start. Capacity climbs in steps, well behind the load. The shaded area between the two lines is traffic with no machine to serve it: queued or failed requests. Three things close that gap. Headroom, so a ramp fits in what is already running. Prediction: if the peak always comes at six, scale at quarter to. And a warm pool of machines already started, which cuts ninety seconds to about ten.
Autoscaling · Going deeper
Reactive, predictive, and their failure modes
Target tracking holds a metric at a set value and is calm in steady state; step scaling adds more machines the further the metric overshoots and reacts faster to a ramp. Scaling on CPU misleads for I/O-bound services. A short cooldown makes the fleet flap; a long one leaves it over-provisioned for hours.
Profiling and flame graphs
A flame graph shows where the time goes: width is time
A profile is a histogram of where time is spent. The widest box is the symptom; the fix is usually in its caller.
The service is slow. Before buying hardware, find out where the time goes. A profiler interrupts the program about ninety-nine times a second and records which functions are running. Stack those samples and you get a flame graph. Each box is a function, sitting on the function that called it. Width is the share of time. The eye goes to the widest box at the top: a reflection call, forty-five percent of all samples. That is the symptom. Reflection being slow is not news. Follow it down to the first function your team owns. Why is it encoding the same five fields fifty thousand times a second? The fix lives there. One warning. A CPU profile sees only time spent running. If the CPU is at twenty percent and requests are still slow, the time is spent waiting, on a lock, a pool, a disk, and you need an off-CPU profile to see it.
Profiling and flame graphs · Going deeper
Reading a profile honestly
CPU time counts cycles on a core; wall time adds everything the thread waited on. Sampling has low bounded overhead; instrumentation is exact but can distort what it measures. Allocation profiles catch memory churn that shows up later as garbage-collection pauses. eBPF-based continuous profilers such as Pyroscope and Parca run for weeks at about 1% overhead.
Percentiles
The average hides the users who wait the longest
Read latency by percentile, not by mean. The tail is where users notice and where the cause shows.
Capacity planning is an estimate. A load test proves it, by driving synthetic traffic at the system before real users do. Here is one result: five hundred requests a second for ten minutes. The median is fifty milliseconds and the mean is sixty. On these bars, drawn to scale, they are slivers. If you stopped here you would call it fast. The ninety-ninth percentile: one request in a hundred takes two hundred milliseconds. The ninety-nine point nine: one in a thousand takes one point four seconds. At five hundred a second, that is thirty users every minute. A sevenfold jump like that points to a discrete event, such as a garbage collection pause. And the maximum, three point eight seconds, happened once in three hundred thousand requests. Note it, but investigate the one that happens hundreds of times.
Percentiles · Going deeper
Four shapes of load test
A steady test holds the target rate as a baseline. A soak holds it for hours or days to expose leaks. A stress test ramps until the system breaks, to find the limit. A spike jumps to five or ten times in one step, to see whether the autoscaler reacts before the objective is breached. Pick the shape that answers the question being asked.
Load testing
A load test that waits for replies hides the delay it measures
Coordinated omission: a closed-loop generator backs off when the server stalls. Test at a fixed arrival rate.
There is a trap in how load tests are run, and it makes bad systems look good. Suppose the server stalls for four seconds. A closed-loop generator sends a request, waits for the reply, then sends the next. So during the stall it sends exactly one request and waits. The three requests it would have sent are never sent. It records one slow reply out of eight. An open-loop generator sends on a schedule, whatever the server is doing, as real users do. Four requests arrive during the stall and all four wait. It records four slow replies out of eleven. Same server, same stall, and the first tool reports a much cleaner result. It coordinated with the server to omit the bad samples.
Load testing · Going deeper
Coordinated omission
Named by Gil Tene. A closed-loop generator keeps a fixed number of virtual users, so its request rate falls whenever the server slows, and the latency histogram is biased by the latency it was meant to measure. wrk2 introduced the corrected constant-rate model; k6 and vegeta support it. Re-run a suspicious result open-loop: if p99.9 jumps from 1.4 s to 5 s, the first run was measuring the generator.
Check yourself
This service uses 20% of its CPU, yet its slowest requests take seconds. Where should you look first?
Low CPU with slow requests means the time is spent blocked, not computing. A CPU profile cannot see waiting; an off-CPU profile or a trace can.
Tracing and sampling
Too many traces to keep: decide which after they finish
A trace ties one request’s spans together across services. Tail sampling keeps the slow and failed ones.
A user reports that one checkout took eight seconds. The dashboards look fine, because that request is one in millions, and it crossed seven services. A trace is the record that survives this: every service passes along a trace id and records a timed span, and the spans reassemble into the story of that one request. The difficulty is volume. Here are sixty requests; one is the slow one. Keeping a trace for every request costs as much as the system itself. The cheap answer is head sampling: flip a coin when the request arrives and keep one percent. But the decision is made before anyone knows which request is interesting, so the slow one is almost always thrown away. Tail sampling waits. A collector holds the spans until the request ends, then keeps every slow or failed one and a small sample of the rest. That is what an investigator wants, paid for with memory.
Tracing and sampling · Going deeper
Spans, context and exemplars
A span is one unit of work in one service, linked to its parent. W3C Trace Context standardises the traceparent header that carries the ids. Exemplars attach a few trace ids to each histogram bucket, so a click on a slow bar of a latency chart opens a real trace from it. OpenTelemetry is the vendor-neutral way to instrument once and change backends later.
SLOs and burn-rate alerts
Page on how fast the error budget burns, not on each error
Burn rate is how many times faster than planned the error budget is being spent. Two rates, two alerts.
The last lesson introduced the error budget. A target of ninety-nine point nine percent over thirty days allows forty-three minutes of failure. The operational question is when to wake someone. Spend the budget at exactly the planned pace, a burn rate of one, and it lasts the full thirty days. At six times the pace it is gone in five days. At fourteen point four times, in two days. So there are two alerts. The fast one pages when both the last hour and the last five minutes are above fourteen point four: that is a bad release. The slow one pages when the last six hours and the last thirty minutes are above six: that is a leak. The long window ignores a thirty-second blip. The short window clears the page as soon as the fix works.
SLOs and burn-rate alerts · Going deeper
Multi-window, multi-burn-rate alerts
From the Google SRE Workbook. 14.4× is chosen because it spends 2% of a monthly budget in one hour. A naive alert on the current error rate pages on every blip and misses a slow leak that eats the whole month. Higher targets cost more: 99.99% needs multi-region replication and automated failover. An SLO is only a contract if exhausting the budget really does stop releases.
Check yourself
The target allows 0.1% of requests to fail over 30 days. Right now 0.5% are failing. If nothing changes, when is the budget gone?
Burn rate is observed ÷ allowed: 0.5 ÷ 0.1 = 5×. A 30-day budget spent five times too fast lasts 30 ÷ 5 = 6 days.
Chaos engineering
Break it on purpose, small, before it breaks by itself
Chaos engineering: a controlled experiment with a hypothesis and a bounded blast radius. It tests the runbook.
A system that has never failed in production has only not failed yet. And the runbook for a failure that has never happened is fiction until someone tries to use it. So try it deliberately. Start with a hypothesis: we believe retries will absorb a slow payment service. Inject the fault, five hundred milliseconds of latency for ten minutes, inside a strict boundary: one pod, a time limit, a stop button. Grow that boundary only as trust is earned, from one pod outside production, to one zone, to the whole fleet. Watch what happens, and ship what you learn: a fix, a corrected runbook, and a repeat of the test. What experiments find is rarely what the team expected. Retries that pile up and make things worse. A connection pool pinned to a dead replica for five minutes.
Chaos engineering · Going deeper
Anatomy of an experiment
Faults fall into a short list: network (latency, loss, partition, DNS), resource (CPU, memory, disk), dependency (errors, slowness, throttling), process kills, and zone or region loss. Game days are scheduled exercises; continuous chaos keeps probing resilience already built. Chaos Mesh and Litmus inject faults on Kubernetes; cloud providers offer their own.
RPO and RTO
Recovery has two numbers: data lost and time down
RPO is the data you can afford to lose; RTO is the downtime. Replication sets the first, rehearsal the second.
A backup is a file. A disaster needs a working service. Follow one regional failover on a clock. At zero, the primary region fails. Look to the left first. The standby had received every write up to eight seconds before the failure. Those eight seconds of writes are gone. That is the recovery point: how much data is lost. Now to the right. Thirty seconds for health checks to detect it. Another thirty for a person to decide. Then promote the copy, switch the address, wait for traffic to move, and warm the cold caches. Normal service at four minutes thirty. That is the recovery time: how long you were down. Both numbers were promised in advance, and both were decided by design: how the data is copied, how the address is switched, and whether anyone had practised.
RPO and RTO · Going deeper · 1 of 2
What sets each number
Synchronous replication gives a recovery point of zero and charges every write a cross-region round trip. Asynchronous replication loses whatever the lag was, which grows under load. Recovery time is the sum of detection, decision, promotion and traffic redirection: a 60-second DNS lifetime alone adds a minute. Idempotency keys make client retries safe during the switch.
RPO and RTO · Going deeper · 2 of 2
Three recovery designs
Active-passive keeps a standby that takes over on a switch: cheaper, slower. Active-active serves from both sides and simply loses one: fast, costly, and writes must converge. Cell-based design splits customers into independent full stacks, so one failure reaches only one cell’s customers.
Incident response
In an incident, stop the harm first and find the cause later
Incident response: a named commander, one channel, and mitigation before diagnosis. The review comes after.
Production will break. What matters is how the team behaves in the next thirty minutes. The first ten run to a checklist. The page fires. Someone acknowledges within five minutes, or it escalates. By minute eight a first public update goes out, two sentences long. In between: open one channel, state the symptom in one sentence, name the roles, and start mitigating. The commander coordinates and does not debug. The rule that matters most is to mitigate before you understand. Roll the release back, turn the flag off. Users recover in minutes, and the cause is investigated afterwards. Debug first, and users stay broken for as long as the search takes. Afterwards, a blameless review asks why, five times, until it reaches the process that allowed it.
Incident response · Going deeper · 1 of 2
Roles in the first ten minutes
The incident commander is a role, not a rank, and rotates with on-call. Updates are templates: “We are investigating an issue affecting checkout. Updates every 15 minutes.” Useful prompts under pressure: what else could it be, what is the worst outcome of rolling back, who else should be here. “Who broke this?” shuts the room down.
Incident response · Going deeper · 2 of 2
Severity levels
SEV1 is a full outage or data-integrity event and pages everyone. SEV2 is a major impairment, SEV3 a minor one handled in working hours, SEV4 a ticket. A humane on-call sees fewer than two pages a week and two night pages a quarter; above that people leave, and the next outage is handled by those who missed the last one.
Summary
Operations is a loop: plan, run, learn
Plan for the peak, watch the tail, learn from every failure. Next: intelligence, how machines learn from data.
Put the lesson together. Plan: size the fleet for the busiest hour and keep it under seventy percent busy, then prove the plan with a load test that holds a fixed arrival rate. Run: scale ahead of the load, trace and profile the slow tail, and page when the error budget burns too fast. Learn: break things on purpose in small ways, rehearse the failover, and review incidents without blame. And what you learn changes the next plan. These three run at the same time, all the time. Next, a different kind of system: one that learns its behaviour from data.
Summary · Going deeper
The practices on one map
Plan, run and learn run in parallel, not in sequence: last quarter’s postmortems inform this quarter’s capacity plan, and this week’s chaos drill exposes next week’s runbook gap. The canonical reading is Google’s Site Reliability Engineering and its Workbook, both free online.
Lesson complete
You finished “Operations”
Answer the questions along the way to see your score here.