Capacity planning

Size for the busiest hour, because the average hides it

00:0006:0012:0018:0024:00average: what the dashboard showspeak: 4× the averagea spike nobody forecastcapacity to provision: peak + spike + six months of growthPeak ÷ averagetypical web service3 – 5×one time zone8 – 10×back office1.5×Growth of 8% a monthdoubles traffic in 9 monthsUsers meet the peak, never the average, so capacity is sized for the worst expected hour.A service at 30% average CPU can still fall over every Friday at 18:00. And growth compounds: at 8% a month, a peak that is60% of capacity today passes 100% in about seven months, so capacity is checked every quarter, not bought once.

Capacity planning provisions for the peak, the spike on top of it, and growth. The average is for the spreadsheet.

Capacity planning · Going deeper

Diurnal load profile — daily peak versus average, with the spike on top 100% 80% 40% 0% 00:00 12:00 24:00 avg 30% peak 80% flash event peak-to-average ratio = 80 / 30 ≈ 2.7× capacity must clear the peak plus headroom plus the unforeseen spike

The load profile

Three numbers describe load: the average, the peak, and their ratio. Compute monthly growth from six months of traffic and project it forward. Under-provisioning costs the service-level objective; over-provisioning costs the cloud bill. The right margin depends on how spiky the traffic is and how fast new capacity can start.

Utilisation and latency

Latency stays flat, then explodes: keep servers under 70% busy

0%50%70%90%100%how busy the server isresponse time, as a multiple of the work itselfpast the knee2×3.3×10×W = 1 / (μ − λ)μ: requests the server can finisheach secondλ: requests arriving each secondAs arrivals approach capacity,the wait has no upper limit.Response time does not rise in a straight line with load. Past about 70% busy it climbs steeply.Requests queue behind each other. At 50% busy a request takes twice its own work time; at 90%, ten times.Running hotter saves on servers and spends the latency budget, so fleets are sized to stay under about 70% at peak.

Queueing makes response time rise hyperbolically with utilisation. About 70% at peak is the practical ceiling.

Utilisation and latency · Going deeper

M/M/1 response time versus utilisation — the knee at around 70% response time 0% 50% 70% 85% 95% 100% the knee around 70% linear region latency ≈ service time non-linear region latency → ∞ as ρ → 1 M/M/1: W = 1 / (μ − λ) · response time blows up as λ approaches μ

The M/M/1 curve

For a single server with random arrivals, response time is W = 1 / (μ − λ), which goes to infinity as utilisation approaches one. Real fleets with many servers have a gentler curve of the same shape. Past the line where arrivals meet service rate there is no graceful degradation: work piles up until something breaks.

Little’s law

Work in flight = arrival rate × time each request spends inside

500 requests / secondarrival rate×0.060 s eachtime spent inside=30 in flightat any instantA pool of 32 workers, each request taking 60 ms30 busy, 2 free: it just fitsSame traffic, but a dependency slows each request to 80 ms: 500 × 0.080 = 408 must waitevery worker busythe queue: waiting makes these slower stillWork in flight = arrival rate × time each request spends inside. Know two and the third is fixed.This is Little’s law, and it holds for web servers, queues and thread pools alike. The 20 ms of slowness added no traffic,yet it filled the pool. Slowness itself creates queues, which is why one slow dependency spreads through a system.

Little’s law: L = λ × W. A dependency that slows down fills your worker pool without one extra request arriving.

Little’s law · Going deeper

Little's law connecting arrival rate, residence time, and in-flight count system L = items inside W = time inside per item λ in arrivals / s λ out departures / s L = λ · W holds for any stable queueing system · 500 RPS × 60 ms = 30 in flight

Little’s law

L = λW holds for any stable queueing system regardless of its internal design: web servers, message queues, thread pools, garbage collectors. Knowing any two quantities fixes the third. An autoscaling threshold is in effect a target W for an expected λ.

Check yourself

200 / secondarriving0.1 seach one takes?in flight

How many requests are inside this service at any instant?

Autoscaling

Autoscaling always arrives late, and users pay for the gap

0 s60120180240 s1020machinesover capacity:queued or failedload doublesin 60 scapacity: each newmachine needs 90 sto startwith a warm pool: ready in 10 sThree ways to close the gapHeadroomrun below 70% so a ramp fitsPredictionscale at 17:45 for the 18:00 peakWarm poolmachines started and waitingAn autoscaler reacts after the load arrives, and new machines take time to start. Users pay for the gap.A 99.99% target allows about 4 minutes of failure a month: roughly one slow scale-up. Scale on requests per machine,which follows user load more closely than CPU does, and treat autoscaling as the executor of a capacity plan, not a substitute.

Autoscaling follows load with a delay. Headroom, prediction and warm pools are what cover the delay.

Autoscaling · Going deeper

Reactive autoscaling lag — load steps up, capacity follows behind RPS t=0 spike caught up demand capacity (lagging) budget-burning gap add headroom equal to (cold-start time × peak ramp rate) to keep the gap from touching the SLO

Reactive, predictive, and their failure modes

Target tracking holds a metric at a set value and is calm in steady state; step scaling adds more machines the further the metric overshoots and reacts faster to a ramp. Scaling on CPU misleads for I/O-bound services. A short cooldown makes the fleet flap; a long one leaves it over-provisioned for hours.

Profiling and flame graphs

A flame graph shows where the time goes: width is time

mainhandleRequestloadOrderdb.queryescapeencodeJSONreflect.Fieldwidest box: 45% of all samples. The symptom.renderResponsefirst function you own: why does it encode the same five fields 50,000 times a second?width = share of timeheight = who called whomWhat a CPU profile cannot seeon CPU 20%waiting 80%Low CPU and slow requests meansthe time is spent blocked: on a lock,a connection pool, a disk flush.An off-CPU profile shows that part.A profiler samples the running program about 99 times a second and stacks up where it was. Width is time.The widest box is where to start asking, not the fix: follow it down to the first function your team owns and ask why it doesthat much work. Continuous profilers now run in production all the time at about 1% overhead.

A profile is a histogram of where time is spent. The widest box is the symptom; the fix is usually in its caller.

Profiling and flame graphs · Going deeper

A flame graph — stacked sample counts, width proportional to time on CPU main setup handleRequest flush parse serialise commit encodeJSON gzip reflect.Value.Field width = fraction of samples · widest bar = where time goes · towers = call depth read top-down: which leaf is fattest, then ask why its caller called it that much

Reading a profile honestly

CPU time counts cycles on a core; wall time adds everything the thread waited on. Sampling has low bounded overhead; instrumentation is exact but can distort what it measures. Allocation profiles catch memory churn that shows up later as garbage-collection pauses. eBPF-based continuous profilers such as Pyroscope and Parca run for weeks at about 1% overhead.

Percentiles

The average hides the users who wait the longest

One load test: 500 requests a second for 10 minutes, no errors. Bars are drawn to scale.median50 ms: half of all requests are faster than thismean60 ms: the number on most dashboardsp99200 ms: 1 request in 100p99.91.4 s: 1 in 1,000, which is 30 users every minutemax3.8 sone request out of 300,000: note it, do not chase itThe average says 60 ms. One user every two seconds waits 1.4 seconds. Latency is read at the tail.A 7× jump from p99 to p99.9 is the mark of a discrete event: a garbage-collection pause, a network retransmit, a wait fora pooled connection. Investigate the percentile that happens hundreds of times, not the single worst request.

Read latency by percentile, not by mean. The tail is where users notice and where the cause shows.

Percentiles · Going deeper

Four load-test shapes — steady, soak, stress, spike steady soak stress spike baseline RPS · 30 min target RPS · 24+ h ramp until failure step to 5× find typical p99 find leaks find breakpoint find scaling lag A load test is only as good as the generator. A closed-loop generator throttles itself when the server slows down, which hides the very tail latency a real client population would inflict on its peers.

Four shapes of load test

A steady test holds the target rate as a baseline. A soak holds it for hours or days to expose leaks. A stress test ramps until the system breaks, to find the limit. A spike jumps to five or ten times in one step, to see whether the autoscaler reacts before the objective is breached. Pick the shape that answers the question being asked.

Load testing

A load test that waits for replies hides the delay it measures

0 s2610 sServerstalls for 4 secondsClosed loopwaits for each replythree requests never sentrecords1 slow of 8Open loopsends on schedulerecords4 slow of 11A generator that waits for each reply slows down when the server does, hiding the very delay it measures.This is coordinated omission. Real users do not wait their turn. Tools that hold a fixed arrival rate (wrk2, k6, vegeta) reportthe true tail; closed-loop tools, the default in many scripts, can understate it by a factor of ten.

Coordinated omission: a closed-loop generator backs off when the server stalls. Test at a fixed arrival rate.

Load testing · Going deeper

Coordinated omission — closed-loop generator misses the tail the server actually produced Closed-loop · 30 virtual users Open-loop · 500 RPS target slow request few samples · generator throttled itself · reported p99 = 80 ms backlog from same slow request backlog billed to client · reported p99 = 800 ms (10× higher)

Coordinated omission

Named by Gil Tene. A closed-loop generator keeps a fixed number of virtual users, so its request rate falls whenever the server slows, and the latency histogram is biased by the latency it was meant to measure. wrk2 introduced the corrected constant-rate model; k6 and vegeta support it. Re-run a suspicious result open-loop: if p99.9 jumps from 1.4 s to 5 s, the first run was measuring the generator.

Check yourself

CPU in use: 20%idle: 80%

This service uses 20% of its CPU, yet its slowest requests take seconds. Where should you look first?

Tracing and sampling

Too many traces to keep: decide which after they finish

All requestsheight = time takenone request took 8 s, across seven servicesHead samplingdecide at the doora coin flip keeps 1%: cheap, and the slow one is almost always thrown awayTail samplingdecide at the endkeeps every slow or failed request, plus a small sample of normal onesA trace follows one request across services. The hard part is keeping the one in a million that matters.Each service passes a trace id along in a header (W3C traceparent) and records a timed span. Storing every trace coststoo much, so a collector holds the spans until the request ends, then decides. The price is memory in the collector.

A trace ties one request’s spans together across services. Tail sampling keeps the slow and failed ones.

Tracing and sampling · Going deeper

A trace — root span, child spans by service, a Gantt-style waterfall 0 ms 100 ms 200 ms edge GET /checkout · 200 ms auth · 20 ms cart svc · 40 ms db SELECT · 25 ms payment · 60 ms stripe API · 45 ms notify · 40 ms parent edge span runs the full 200 ms · children sit below their parent in the tree the longest critical-path child is payment → stripe; optimisation lives there attributes per span: http.status_code, db.statement, tenant.id, user.id (sanitised) propagation: traceparent header on every outbound call carries the trace id forward sampling decision: keep this trace if any span had status=error or duration > 500 ms (tail sampling)

Spans, context and exemplars

A span is one unit of work in one service, linked to its parent. W3C Trace Context standardises the traceparent header that carries the ids. Exemplars attach a few trace ids to each histogram bucket, so a click on a slow bar of a latency chart opens a real trace from it. OpenTelemetry is the vendor-neutral way to instrument once and change backends later.

SLOs and burn-rate alerts

Page on how fast the error budget burns, not on each error

99.9% over 30 days = 43.2 minutes of failure allowed100%0budgetleftday 051020301×: the planned pace, lasts the full 30 days6×: gone in 5 days14.4×: gone in 2 daysFast burn: page1 h and 5 min both above 14.4×catches a bad releaseSlow burn: page6 h and 30 min both above 6×catches a leaking dependencyThe long window ignores blips.The short one clears the pageas soon as the fix works.Alert on how fast the error budget is burning, not on the error count at this moment.Burn rate = observed error rate ÷ allowed error rate. A 30-second blip moves no window enough to page. A leak of 0.7% errorsnever looks dramatic, yet burns at 7× and is caught by the slow alert. The budget only works if exhausting it halts releases.

Burn rate is how many times faster than planned the error budget is being spent. Two rates, two alerts.

SLOs and burn-rate alerts · Going deeper

Multi-window burn-rate alerting — short and long windows, two thresholds burn rate 14.4× 6× 1× t=0 2 d 30 d fast page slow page steady fast burn · 1h/5m → page in minutes slow burn · 6h/30m → page in tens of minutes page only when long window AND short window both exceed the threshold short window prevents stale paging on a fixed problem · long window prevents flapping

Multi-window, multi-burn-rate alerts

From the Google SRE Workbook. 14.4× is chosen because it spends 2% of a monthly budget in one hour. A naive alert on the current error rate pages on every blip and misses a slow leak that eats the whole month. Higher targets cost more: 99.99% needs multi-region replication and automated failover. An SLO is only a contract if exhausting the budget really does stop releases.

Check yourself

allowed0.1%failing0.5%

The target allows 0.1% of requests to fail over 30 days. Right now 0.5% are failing. If nothing changes, when is the budget gone?

Chaos engineering

Break it on purpose, small, before it breaks by itself

Hypothesisretries absorb aslow payment serviceFaultadd 500 ms latencyfor 10 minutesBlast radiusone pod, a time limit,a stop buttonObservedashboards, alerts,support channelLearna fix, a correctedrunbook, a repeat testGrow the blast radius only as trust is earnedwhole fleet, continuouslyone zone in production, fixed timeone pod, outside productionWhat experiments keep findingRetries absorb a slow dependencyretries pile up, making it worseReads fail over to the replicapool pinned to dead replica 5 minThe autoscaler covers a spikecold start too slow for the spikethe team believedthe experiment showedInject a small, bounded failure on purpose, with a stated prediction, while the team is awake.Netflix began in 2010 by killing random servers during working hours. The value is rarely the fault itself: it is the gap itexposes in a runbook nobody had tried. Without a hypothesis and a stop condition it is an incident with a calendar invite.

Chaos engineering: a controlled experiment with a hypothesis and a bounded blast radius. It tests the runbook.

Chaos engineering · Going deeper

A chaos experiment — hypothesis, fault, blast radius, observation, learning 1. hypothesise 2. inject fault 3. observe 4. learn retries handle 500ms latency on payment service tc qdisc netem +500ms · 1 az · 10 min · canary SLO dashboard runbook · alerts user complaints update runbook file tickets repeat next mo Fault taxonomy network · latency, packet loss, partition, DNS failure resource · CPU starvation, memory pressure, disk fill, fd exhaustion dependency · downstream errors, throttling, slow response, partial outage application · process kill, OOM, malformed config reload, clock skew infrastructure · az failure, region failure, control-plane outage always set a blast radius: smallest scope · time-boxed · with an abort button

Anatomy of an experiment

Faults fall into a short list: network (latency, loss, partition, DNS), resource (CPU, memory, disk), dependency (errors, slowness, throttling), process kills, and zone or region loss. Game days are scheduled exercises; continuous chaos keeps probing resilience already built. Chaos Mesh and Litmus inject faults on Kubernetes; cloud providers offer their own.

RPO and RTO

Recovery has two numbers: data lost and time down

primary region failslast write copied8 s beforeRPO: data lost8 s of writes(promised: under 60 s)detect0:30decide1:00promote copy1:30switch DNS2:00traffic moved3:00warm caches3:30normal4:30RTO: time down 4 min 30 s (promised: under 5 min)5:00Synchronous copy: no data lost, every write waits 10 – 60 ms.Asynchronous copy: you lose the lag, seconds on a good day.A standby region costs about 1.7× one region; two active regions about 2×.RPO is how much data you may lose. RTO is how long you may be down. Both are fixed by design, long before.Recovery is a chain: detect, decide, promote the copy, redirect traffic, warm up. The human decision is the step most likelyto stall. A failover that has not been rehearsed this quarter should be assumed not to work.

RPO is the data you can afford to lose; RTO is the downtime. Replication sets the first, rehearsal the second.

RPO and RTO · Going deeper · 1 of 2

RPO and RTO measured around a disaster event last successful replication disaster service back RPO · data lost RTO · time down sync replication: RPO = 0 · async: RPO = replication lag RTO = detect + decide + DNS/LB cut + warm + drain queues RPO 0 / RTO 0 = active-active, sync write quorum, anycast RPO 1 min / RTO 5 min = active-passive, async stream, automated failover RPO 1 h / RTO 4 h = nightly snapshot + WAL ship, manual failover

What sets each number

Synchronous replication gives a recovery point of zero and charges every write a cross-region round trip. Asynchronous replication loses whatever the lag was, which grows under load. Recovery time is the sum of detection, decision, promotion and traffic redirection: a 60-second DNS lifetime alone adds a minute. Idempotency keys make client retries safe during the switch.

RPO and RTO · Going deeper · 2 of 2

Three DR topologies — active-passive, active-active, cell-based active-passive active-active cell-based active standby async or sync replication RPO: seconds RTO: minutes cost: 1.7× active active sync quorum or CRDT writes RPO: 0 RTO: seconds cost: 2.0× cell A cell B cell C cell D cell E cell F customer → cell routing layer blast radius: one cell cell-based bounds the blast radius without paying the active-active write-convergence tax but multiplies operational surface — N cells means N versions of every dashboard

Three recovery designs

Active-passive keeps a standby that takes over on a switch: cheaper, slower. Active-active serves from both sides and simply loses one: fast, costly, and writes must converge. Cell-based design splits customers into independent full stacks, so one failure reaches only one cell’s customers.

Incident response

In an incident, stop the harm first and find the cause later

page fires0 minacknowledged, or it escalates5first public update810 minOpen one channelall talk in one placeState the symptomone pinned sentenceName the rolescommander · tech lead · commsMitigateroll back, flag off, shift trafficMitigate firstimpactusers recovered · the cause is investigated afterwards, calmlyDebug firstusers affected for as long as the team searches for the causeThe commander coordinates and does not debug. Severity sets who is paged: SEV1 wakes leadership, SEV3 waits for Monday.Stop the harm first; find the cause later. A bad release is rolled back before anyone asks why it was bad.The first ten minutes are a checklist, not a discussion. Afterwards a blameless review asks why five times, down to theprocess that allowed the failure, because “be more careful” prevents nothing.

Incident response: a named commander, one channel, and mitigation before diagnosis. The review comes after.

Incident response · Going deeper · 1 of 2

Incident-response roles and the first-ten-minutes pattern Incident commander Technical lead Comms lead declares · runs cadence decides escalation decides resolved investigates · mitigates runs commands owns root cause external updates internal updates status page First 10 minutes t+0:00 page fires · on-call ack within 5 min t+1:00 declare incident · open channel #inc-2026-05-15-checkout t+2:00 assign IC, technical lead, comms lead (can be one person early on) t+3:00 state the symptom in one sentence · pin to the channel t+5:00 start mitigation: roll back, feature flag off, throttle, drain t+8:00 first external update on status page · template, not prose t+10:00 second decision: scaled wider? escalated? do not chase root cause yet mitigate before resolve · resolve before postmortem · postmortem before next time

Roles in the first ten minutes

The incident commander is a role, not a rank, and rotates with on-call. Updates are templates: “We are investigating an issue affecting checkout. Updates every 15 minutes.” Useful prompts under pressure: what else could it be, what is the worst outcome of rolling back, who else should be here. “Who broke this?” shuts the room down.

Incident response · Going deeper · 2 of 2

Severity scale and the response each level triggers SEV1 SEV2 SEV3 SEV4 full outage data loss many users page all wake leads comms 15 min postmortem in 5 days major impair. key feature significant user page on-call + team lead comms 30 min postmortem in 10 days minor impair. degraded UX some users business hours team channel no exec ping ticket + brief review cosmetic self-healed no user impact ticket only no page no postmortem

Severity levels

SEV1 is a full outage or data-integrity event and pages everyone. SEV2 is a major impairment, SEV3 a minor one handled in working hours, SEV4 a ticket. A humane on-call sees fewer than two pages a week and two night pages a quarter; above that people leave, and the next outage is handled by those who missed the last one.

Summary

Operations is a loop: plan, run, learn

PlanSize for the peak, under 70% busyLoad test at a fixed arrival rateRunScale ahead of the loadTrace and profile the slow tailPage on budget burn rateLearnBreak things on purpose, smallRehearse failover: RPO and RTOReview incidents without blamewhat was learned changes the next planOperations is a loop: plan for the worst hour, watch the tail while it runs, turn every failure into a change.The practice is written down: Google’s SRE book and workbook, the OpenTelemetry specification, W3C Trace Context.

Plan for the peak, watch the tail, learn from every failure. Next: intelligence, how machines learn from data.

Summary · Going deeper

Operations practices and the loops connecting plan, run, and learn PLAN RUN LEARN capacity · perf budget load test · DR design SLOs · observability auto-scale · alerting incident · postmortem chaos · runbook postmortems feed next quarter's capacity plan the operations loop forecast peak load · provision with headroom · prove the headroom with a load test instrument every request · publish an SLO · alert on burn rate, not raw error count break things on purpose during business hours · find the runbook gaps before they find you when something does break, mitigate first, resolve second, learn third — write it down

The practices on one map

Plan, run and learn run in parallel, not in sequence: last quarter’s postmortems inform this quarter’s capacity plan, and this week’s chaos drill exposes next week’s runbook gap. The canonical reading is Google’s Site Reliability Engineering and its Workbook, both free online.

Lesson complete

You finished “Operations”

Answer the questions along the way to see your score here.

Open in a new tab