RPO and RTO
Recovery has two numbers: data you may lose, time you may be down
RPO is the data you may lose, measured as time; RTO is how long you may be down. A daily backup means up to 24 hours of loss.
Every recovery plan starts from two numbers. A shop backs up its database once a day, at midnight. At twenty to six in the evening, the region it runs in is lost. Everything written since midnight is gone: seventeen hours and forty minutes of orders. The most data you are allowed to lose, measured as time, is the recovery point objective. Then it takes four hours to rebuild somewhere else. The longest you are allowed to be down is the recovery time objective. The business sets both, from what lost data and lost hours cost, and every design choice in this lesson buys a smaller number with money.
High availability and disaster recovery · Going deeper
High availability survives a failed part; recovery survives losing a region or your data
Multi-AZ covers servers and zones. Region loss needs copies elsewhere; deleted data and ransomware need backups in a locked vault.
Disaster recovery strategies
Four strategies trade cost against recovery time
Backup and restore, pilot light, warm standby, multi-site active/active: more running in the second region means faster recovery and higher cost.
There are four standard strategies, and they differ in how much already runs in the recovery region. Backup and restore: only copies of data wait there. Recovery means building everything, which takes hours, but it costs little. Pilot light: the database is replicated and live, but the servers are switched off, ready to start. Tens of minutes. Warm standby: a smaller, complete, working copy runs all the time, and recovery means scaling it up. Minutes. Multi-site active, active: both regions serve users all the time, so losing one costs almost nothing, except the price of running everything twice. Choose the cheapest one that meets each workload’s numbers.
Replication and backups
Replication copies your mistakes; only a backup goes back in time
A replica applies a bad change within a second. Point-in-time recovery restores to just before it, into a new database.
Here is the mistake many designs make. At ten o’clock, a bad migration drops the orders table on the primary database. One second later, the cross-region replica drops it too. That is its job: to be an exact copy. Only the backups can help. Point-in-time recovery replays the change log up to one second before the mistake, into a new database beside the old one. So replication protects against losing a place; backups protect against losing the data itself. You need both.
AWS Backup · Going deeper
Copy backups to another region and account, lock them, and test restores
AWS Backup copies across regions and accounts; Vault Lock stops early deletion; restore testing proves the copies work.
Cross-region failover
Fail over the data first, then the traffic
Aurora Global Database replicates in about a second and promotes in about a minute; Route 53 failover moves users when a health check fails.
Here is a warm standby in two regions. Route 53 sends all users to the primary region and checks its health. Data flows to the recovery region all the time. Aurora Global Database copies at the storage layer, typically less than a second behind, and S3 replicates the files. When the primary fails, promote the secondary database to writer, which takes about a minute, scale up the app servers, and Route 53, seeing the failed health check, starts answering with the recovery region.
DNS failover timing · Going deeper
DNS failover takes detection plus the record’s TTL
Detection (30 s × 3) plus a 60 s TTL is about 150 s. Keep failover records’ TTL short.
Testing recovery
A recovery plan you have not run is a guess
This drill took 41 minutes against a 30-minute RTO. The slow steps were deciding and scaling, so pre-agree the trigger and keep capacity warm.
Plans look fast on paper. Here is a real drill. The alarm and the page take five minutes. Then people spend fifteen minutes agreeing to fail over. Promoting the database takes one. Starting enough app servers takes twelve. DNS three, and checking that orders work, five. Forty-one minutes, against a target of thirty. The database was never the problem. The fixes are a trigger agreed in advance, runbooks written as scripts, and capacity that is already running. And run the drill again, regularly, because systems change and the numbers drift.
Lesson complete
You finished “Disaster recovery”
Nicely done.