Vertical scaling
Scaling up is simple until you reach the largest machine
Scaling up means a bigger machine: twice the size, twice the price, a restart to change, and a top size.
The first way to handle more load is a bigger machine. One family of servers comes in sizes that double: two processors and eight gigabytes, then four and sixteen, then eight and thirty-two, all the way up. The price doubles with each size, so the cost of a unit of work stays the same. Moving up needs a restart, but no change to the application. And then there is the largest size. Past it, scaling up is over. The server is also still one server, so when it fails, all of it fails.
Horizontal scaling
Many small servers grow without a ceiling, and a failure costs one share
Scaling out adds identical servers behind a load balancer; losing one of 8 costs 12.5% of capacity.
The second way is more machines. One large server is simple, but when it fails, every user is affected until it is back. Eight small servers behind a load balancer do the same work. When one fails, an eighth of the capacity goes, and the other seven carry on. Need more? Add a ninth. There is no top size. Servers can be updated one at a time with no downtime. The catch: the application must let any server answer any request.
Stateless services
Move the state out, and any server can be copied or thrown away
A stateless server keeps nothing that must outlive a request: data, files and sessions live in services built to hold them.
So what stops a server being copied? State: anything that must still be there for the next request. A typical first server holds it all: the application, the logged-in sessions, uploaded files and the database. Move the database to a managed service. Move uploaded files to object storage. Move sessions to a shared cache. What is left on the server is only the application, the same on every copy. Now it can run as two servers or two hundred, and any one can be replaced without anyone noticing.
Sessions
Keep sessions off the server: in a shared store or a signed token
Sticky sessions pin users to a server; a shared session store or a signed token lets any server answer.
Logins are the state people forget. Three ways to handle them. Sticky sessions: the load balancer sends each user back to the same server, which holds their session in memory. No code changes, but lose that server and its users are logged out, and the load becomes uneven. A shared session store: every server reads sessions from a fast shared cache, so any server can answer. A signed token: the session travels with the user, in a token the servers can verify by its signature. Nothing is stored server-side, though a token is hard to cancel before it expires.
Auto scaling
Auto scaling follows the load between a floor and a ceiling
An Auto Scaling group keeps servers near a target metric, between a minimum and a maximum; here it saves 45% of server-hours.
Load changes through the day, and a fleet sized for the evening peak is mostly idle at night. Here is a day of traffic. An auto scaling group follows it. It has a rule: keep processor use near fifty percent. As load rises it adds servers, as load falls it removes them. It never goes below a minimum of two, one in each zone, or above a maximum of twenty. The alternative is 17 servers all day. The scaled fleet uses 45% fewer server-hours. The same group also replaces any server that fails its health check.
Scaling policies · Going deeper · 1 of 3
Six ways to decide, and the metric matters most
Target tracking is the default; scheduled and predictive scaling act before load arrives; queue workers scale on backlog per server.
Target tracking · Going deeper · 2 of 3
Target tracking is one line of arithmetic: servers × metric ÷ target
A target tracking policy scales in proportion: 8 servers at 80% with a target of 50% become 13.
Auto scaling on three clouds · Going deeper · 3 of 3
One design, three names: scale set, instance group, Auto Scaling group
An Auto Scaling group, an Azure scale set and a Google managed instance group are the same design: a recipe, a group and a policy.
Scaling lag
Reactive scaling arrives after the load; plan ahead for known peaks
New servers take minutes to notice, boot and warm up. For known peaks use scheduled or predictive scaling.
Auto scaling is not instant. At minute ten, load jumps from six hundred to fifteen hundred requests a second. The capacity line stays flat. It takes about three minutes for the metric to show the rise and the alarm to act, and four more for new servers to boot, start and pass their health checks. For seven minutes users wait. If the peak is known, a schedule, or predictive scaling, adds the servers before it begins. For the unknown ones, keep a little headroom, start servers faster, or put a queue in front of the work.
Scale-in · Going deeper
Scaling in removes a live server: drain it, then let it finish
On scale-in a server drains for 300 s, then waits in a lifecycle hook for up to an hour by default; set both to fit the work.