Availability
Each extra nine allows ten times less downtime
Availability is uptime divided by total time. 99.9% allows 43 min of downtime a month.
Availability is the share of time a system works, and it is counted in nines. Most agreements are measured per month, so look at the middle column. Ninety-nine percent allows more than seven hours down every month. Ninety-nine and a half: three and a half hours. Ninety-nine point nine, the most common target: 43 min. Ninety-nine point nine five: about twenty-two minutes. Four nines: four minutes a month. Five nines: twenty-six seconds, less time than it takes to notice a problem. Each nine is ten times harder than the last. The right target is the lowest one the business can afford.
Service level agreements · Going deeper
Provider SLAs promise more when you spread across zones
An SLA is a promise backed by service credits. One VM is promised about 99.5 to 99.9%; VMs across zones 99.99%.
Error budgets
An SLO turns availability into a budget you can spend
An SLI measures, an SLO targets, an SLA promises. A 99.9% SLO is an error budget of 43 minutes a month.
Three terms with three jobs. The service level indicator is the measurement: good requests divided by all requests. The objective is the team’s target for it: ninety-nine point nine percent over thirty days. The agreement is the contract with customers, set lower, at ninety-nine and a half, so the team has room before credits are owed. The objective implies a budget: one tenth of one percent of the month, 43 minutes. Here it is being spent. A bad deploy on day seven. A zone outage on day twenty, twenty-three minutes. There is still budget left, so the team keeps shipping. When it runs out, new features wait, and reliability comes first.
Parts in series
A chain is less available than its weakest link
Parts that must all work multiply: five parts at 99.9% or better give 99.79%.
Now the arithmetic. A request passes through several parts, and all of them must work. DNS, managed and spread worldwide, is close to a hundred percent. Multiply by the load balancer. Multiply by the web servers. By the database. And by an outside payment service at ninety-nine point nine. The result is 99.79%, about 18 hours 23 minutes down a year: lower than any single part. Every hard dependency on the path costs availability. The design lesson: keep the critical path short, and let anything optional fail on its own.
Redundancy
Copies side by side multiply up, if they fail independently
With independent copies the system fails only if all fail: two at 99% give 99.99%.
Redundancy runs the arithmetic the other way. One server at ninety-nine percent is down three and a half days a year. Add a second copy behind a load balancer. Now the system is down only when both are down at the same moment: one percent of one percent. Ninety-nine point nine nine. Three copies: six nines. But this only holds if the copies fail independently. Two servers in the same zone, or running the same faulty release, fail together. Redundancy has to be spread across the thing that fails.
Correlated failure · Going deeper
Copies that share something fail together
Racks, zones, releases, dependencies, regions and accounts are fault isolation boundaries: spread copies across them.
Single points of failure
Find every part that exists only once
A single point of failure is any part whose loss stops the system. Walk the request and ask of each part: what if it disappears?
Before adding copies, find what needs them. Walk the request from the user to the data. DNS: managed DNS is spread worldwide, so that one is usually fine. A single NAT gateway: if its zone fails, every private server loses the internet. One database with no standby: a crash means an outage and a restore. Every server in one zone. Sessions kept in one server’s memory. And the one that is never drawn: a single person who holds the keys. Each is either doubled, or written down as an accepted risk with its recovery time.
Failover
A standby helps only if the switch to it is quick and automatic
Failover time is detection plus promotion plus redirecting clients, and all of it is downtime.
A standby database is not availability by itself. Watch what happens when the primary fails. The failure. First it must be noticed: health checks fail several times in a row, perhaps thirty seconds. Then the standby is promoted to primary. Then clients must find it: a DNS name moves and connections are re-opened. Service returns after about a minute and a half. Managed databases publish these figures: one to two minutes for a classic standby, under thirty-five seconds with two readable standbys, about thirty for Aurora. Choose the one your budget allows, and test it.
Active-active and active-passive · Going deeper
Stateless tiers run all copies at once; databases usually cannot
Active-active uses every copy; active-passive keeps a standby, because writes must agree.
Zone capacity
To survive a lost zone, the others must already carry the load
With 12 servers needed, two zones must run 24 and three zones 18: spare capacity is the price of availability.
One last piece of arithmetic. The peak needs 12 servers. Spread them over two zones and lose one, and the other must carry everything. So each zone needs 12: 24 in all, double the need. Spread over three zones, losing one leaves two, so each needs 6: 18 in all, only half as much again. The servers are already running when the zone fails. Nothing has to scale up in the middle of an incident, when the provider’s own systems may be struggling too. That is called static stability.