Vertical scaling

Scaling up is simple until you reach the largest machine

One family, nine sizes (M7i general purpose)vCPUs, to scalem7i.large2 vCPU · 8 GiB$0.10 an hourm7i.xlarge4 vCPU · 16 GiB$0.20 an hourm7i.2xlarge8 vCPU · 32 GiB$0.40 an hourm7i.4xlarge16 vCPU · 64 GiB$0.81 an hourm7i.8xlarge32 vCPU · 128 GiB$1.61 an hourm7i.12xlarge48 vCPU · 192 GiB$2.42 an hourm7i.16xlarge64 vCPU · 256 GiB$3.23 an hourm7i.24xlarge96 vCPU · 384 GiB$4.84 an hourm7i.48xlarge192 vCPU · 768 GiB$9.68 an hourthe largest: there is no next size upScaling up is the simplest fix, until you reach the largest machine.Each size doubles the processors and memory and doubles the price, so cost per unit of work stays flat. Moving to abigger size needs a stop and a start, and the server is still one server: one failure takes all of it. Scale up for adatabase or a quick fix; scale out for anything that must keep growing. Prices: example list prices, us-east-1, 2025.

Scaling up means a bigger machine: twice the size, twice the price, a restart to change, and a top size.

Horizontal scaling

Many small servers grow without a ceiling, and a failure costs one share

Up: one large serverit fails:100% of capacity goneuntil it is replacedOut: 8 small servers behind a balancerone fails: 12.5% gone; the other 7 carry onMany small servers grow without a ceiling, and a failure costs only one share.Horizontal scaling adds identical servers behind a load balancer. Capacity grows in small steps, the fleet survivesthe loss of any member (here 12.5% of capacity), and servers can be replaced or updated one at a time. The price isthat the application must allow it: any server must be able to answer any request. That is the next page.

Scaling out adds identical servers behind a load balancer; losing one of 8 costs 12.5% of capacity.

Stateless services

Move the state out, and any server can be copied or thrown away

1. Everything on one serverserverappsessionsuploadsdatabase2. Database to a managed serviceserverappsessionsuploadsdatabase3. Files to object storageserverappsessionsdatabaseuploads4. Sessions to a shared cachestateless serverappdatabaseuploadssessions× nMove the state out of the server, and the server can be copied, replaced or thrown away.State is anything that must survive the next request: data, files, sessions. While it lives on the server, that serveris special and cannot be copied. Move each kind to a service built to hold it (a managed database, object storage, ashared cache), and what is left is a stateless server: identical, disposable, and safe to run by the dozen.

A stateless server keeps nothing that must outlive a request: data, files and sessions live in services built to hold them.

Sessions

Keep sessions off the server: in a shared store or a signed token

Sticky sessionsholds the session✓no change to the application✗server lost: user logged out; load unevenShared session storesessions✓any server, any request✗one more service to run and keep availableSigned tokentoken: user 42✓nothing stored on the server side✗hard to revoke before it expiresKeep sessions off the server: in a shared store, or in a signed token the client carries.Sticky sessions (session affinity) pin a user to one server; it works without code changes but breaks the promisethat any server can be lost. A shared store such as Redis or Valkey (ElastiCache, Azure Cache, Memorystore) or adatabase table lets any server answer. A signed token such as a JWT carries the session itself, verified by its signature.

Sticky sessions pin users to a server; a shared session store or a signed token lets any server answer.

Auto scaling

Auto scaling follows the load between a floor and a ceiling

00:0006:0012:0018:0024:0001,0002,000requests a secondload17 servers at 19:00floor: 2 serversfixed fleet sized for the peak: 17 servers all dayAuto scaling follows the load between a floor and a ceiling. Here it runs 45% fewer server-hours.The group keeps each server near a target (50% processor, about 100 requests a second), adding servers as load rises andremoving them as it falls, never below the minimum of 2 (one per zone) nor above the maximum of 20. A fleet sized for thepeak would run 408 server-hours a day; the scaled fleet runs 224. The same group also replaces any server that fails.

An Auto Scaling group keeps servers near a target metric, between a minimum and a maximum; here it saves 45% of server-hours.

Scaling policies · Going deeper · 1 of 3

Six ways to decide, and the metric matters most

PolicyHow it decidesUse it forTarget trackingkeep a metric near a value: “CPU at 50%”the default choice; it adds and removes for youStep scalingadd more servers the further past an alarmlarge, sudden changes; fine controlSimple scalingone alarm, one change, then a cooldownolder; prefer the two aboveScheduledset capacity at a time: “20 at 08:45 on weekdays”load you can predict from the calendarPredictivelearns daily and weekly patterns, scales aheadregular cycles with a slow start-upOn a queuemessages waiting ÷ servers = backlog per serverworkers that read from a queue (lesson 7)Choose what to scale on as carefully as how: the metric must rise when the servers are the bottleneck.Web tiers scale on processor use or requests per target; queue workers on the backlog per server, because CPU stayslow while messages pile up. A cooldown or warm-up stops the group reacting to servers that have not started yet.The same ideas, with different names: EC2 Auto Scaling groups, Azure Virtual Machine Scale Sets with autoscale rules,Google managed instance groups with an autoscaler.

Target tracking is the default; scheduled and predictive scaling act before load arrives; queue workers scale on backlog per server.

Target tracking · Going deeper · 2 of 3

Target tracking is one line of arithmetic: servers × metric ÷ target

$ aws autoscaling put-scaling-policy \ --auto-scaling-group-name web \ --policy-name cpu50 \ --policy-type TargetTrackingScaling \ --target-tracking-configuration file://cpu50.jsoncpu50.json{ "TargetValue": 50.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ASGAverageCPUUtilization" } }What the group then works out8 servers at 80% CPU8 × 80 ÷ 50 = 12.8, so 13 servers, now at 49%night, 20% CPU: 13 × 20 ÷ 50 = 5.2, so 6 serversout when any policy asks; in only when all agreeservers still warming up are not countedTarget tracking sizes the group in proportion: servers × metric ÷ target, rounded up.So the metric must fall as servers are added. Average CPU and requests per target do; total requests and queue length donot (divide the backlog by the fleet first). Ready-made metrics: ASGAverageCPUUtilization, ASGAverageNetworkIn andOut, ALBRequestCountPerTarget. AWS states the proportional rule, not the exact algorithm; never edit the alarms it creates.

A target tracking policy scales in proportion: 8 servers at 80% with a target of 50% become 13.

Auto scaling on three clouds · Going deeper · 3 of 3

One design, three names: scale set, instance group, Auto Scaling group

AWSAzureGoogle CloudThe groupAuto Scaling groupVirtual Machine Scale Setmanaged instance group (MIG)Machine recipelaunch templatescale set model: image and sizeinstance templateFollow a metrictarget tracking; step scalingautoscale rules, each with a cooldowntarget utilisation: CPU, load, metricFollow the calendarscheduled actionsrecurrence and fixed-date profilesscaling schedulesScale aheadpredictive scalingpredictive autoscale: CPU, out onlypredictive autoscaling: CPU onlyScale in whenevery policy agreesevery scale-in rule is metthe peak of the last 10 minutes allowsReplace a failed VMEC2 or load balancer health checksautomatic instance repairsautohealing with a health checkSpreadacross Availability Zonesacross availability zonesa regional MIG across zonesThree clouds, one design: a recipe, a group, a policy, and caution when shrinking.All three add machines as soon as any rule asks and remove them only when every rule allows it: too few servers costsusers, too many costs a few minutes of rent. Azure’s predictive autoscale needs seven days of history and only scalesout. Google sizes the group for the highest load of the last 10 minutes, so a short dip does not remove machines.

An Auto Scaling group, an Azure scale set and a Google managed instance group are the same design: a recipe, a group and a policy.

Scaling lag

Reactive scaling arrives after the load; plan ahead for known peaks

0 min10 min20 min30 minloadcapacityoverloaded3 min to notice4 min to boot and warm upscheduled or predictive: capacity is ready before the load arrivesReactive scaling arrives after the load. Here users wait 7 minutes; plan ahead, or keep a buffer.Auto scaling sees the load only through a metric, then launches servers that must boot, start the application and passhealth checks. For a sudden spike that is several minutes of overload. Fixes: schedule capacity for known peaks, usepredictive scaling, keep headroom with a lower target, start faster (smaller images, warm pools), or queue the work.

New servers take minutes to notice, boot and warm up. For known peaks use scheduled or predictive scaling.

Scale-in · Going deeper

Scaling in removes a live server: drain it, then let it finish

One server leaving the group, with the defaults, to scalelifecycle hook, Terminating:Wait: your shutdown script runs, for up to 3,600 s unless it reports donechosen300 s1,800 s3,900 s: terminatedoff the load balancer and draining: 300 sWhich server goes (default termination policy)1. one in the zone that has the most servers2. then one on an old launch template or version3. then the one closest to its next billing hourWhat you setderegistration delay: seconds, not minutes, for short requestshook timeout: AWS advises 30 to 120 s for scale-inscale-in protection: a busy worker cannot be chosenSpot: about 2 minutes’ notice, whatever the hook saysScale-in removes a server that is still serving: drain it, let it finish, then terminate it.Left at the defaults, a server can linger 65 minutes after the decision: 300 s draining connections, then up to an hourin the hook. Set both to what the work needs. A web server stops taking requests and ships its logs; a queue workerfinishes or releases its message. Keeping zones balanced always comes before the termination policy.

On scale-in a server drains for 300 s, then waits in a lifecycle hook for up to an hour by default; set both to fit the work.

Lesson complete

You finished “Scaling”

Nicely done.

Open in a new tab