RPO and RTO

Recovery has two numbers: data you may lose, time you may be down

00:0006:0012:0018:0024:00last backup 00:00taken once a dayregion lost 17:40data written since the backup is gone: 17 h 40 minRPO, recovery point objective: the most data you may lose, as timedown 4 hback 21:40RTO, recovery time objective: the longest you may be downRecovery has two numbers: how much data you may lose (RPO) and how long you may be down (RTO).A daily backup means a loss of up to 24 hours; this disaster cost 17 h 40 min of orders, then 4 hours rebuilding.The business sets both numbers per workload, from what an hour of lost data or downtime costs. RPO decides how oftendata is copied away; RTO decides how much must already be running elsewhere. Tighter numbers cost more.

RPO is the data you may lose, measured as time; RTO is how long you may be down. A daily backup means up to 24 hours of loss.

High availability and disaster recovery · Going deeper

High availability survives a failed part; recovery survives losing a region or your data

What goes wrongMulti-AZ standbyCross-region replicaBackupsA server fails✓✓✓ (slow)A whole zone fails✓✓✓ (slow)A whole region fails✗✓✓ if copied to another regionA bad change deletes data✗ copies the delete✗ copies the delete✓ restore to before itRansomware or a stolen admin key✗✗✓ locked, in another accountHigh availability survives a failed part. Disaster recovery survives losing a region, or your own data.Multi-AZ keeps running through server and zone failures in seconds or minutes, but it lives in one region and copiesevery change, including mistakes. Recovery from a region loss needs copies elsewhere; recovery from corruption oran attacker needs copies that go back in time and that the attacker cannot reach. Most designs need all three.

Multi-AZ covers servers and zones. Region loss needs copies elsewhere; deleted data and ransomware need backups in a locked vault.

Disaster recovery strategies

Four strategies trade cost against recovery time

Backup and restorerecovery regiononly copies wait thereRPO and RTO: hoursabout 5% of the primary’s costPilot lightrecovery regiondata live, servers offRPO/RTO: tens of minutesabout 15% of the primary’s costWarm standbyrecovery regionsmall full copy runningRPO/RTO: minutesabout 40% of the primary’s costMulti-site active/activerecovery regionfull copy serving usersRPO/RTO: near zeroabout 100% of the primary’s costFour strategies trade cost against recovery time: the more already runs in the second region, the faster you recover.Backup and restore rebuilds everything from copies. Pilot light keeps the data replicated and the servers ready to start.Warm standby runs a smaller working copy to scale up. Active/active serves users from both regions all the time.Pick the cheapest one that meets each workload’s RPO and RTO (shares of cost are illustrative).

Backup and restore, pilot light, warm standby, multi-site active/active: more running in the second region means faster recovery and higher cost.

Replication and backups

Replication copies your mistakes; only a backup goes back in time

09:3009:4510:0010:1510:30Primary databasetable dropped 10:00:00Cross-region replicadropped too, 10:00:01Point-in-time backupsrestore to 09:59:59 as a new databaseReplication copies your mistakes in a second. Only a backup can take you back to before the mistake.A replica is a live mirror: perfect for a lost server or region, useless when the damage is a deleted table, a badmigration or encrypted files, because it applies the damage too. Point-in-time recovery replays the change log to anysecond in the retention window (RDS and Aurora: up to 35 days), and restores into a new database beside the old one.

A replica applies a bad change within a second. Point-in-time recovery restores to just before it, into a new database.

AWS Backup · Going deeper

Copy backups to another region and account, lock them, and test restores

Production account · us-east-1RDS, DynamoDB,EBS, EFS, S3backup plandaily, keep 35 daysSame accountother regionvault copyus-west-2cross-region copyBackup accountlocked vaultVault Lock: no deletescross-account copyCopy backups to another region and another account, lock them, and test that they restore.AWS Backup applies one plan to many services, copies across regions and accounts, and Vault Lock makes copiesimpossible to delete early, even for an administrator, so a stolen key cannot destroy them. Restore testingrestores samples on a schedule and checks them: a backup you have never restored is only a hope.

AWS Backup copies across regions and accounts; Vault Lock stops early deletion; restore testing proves the copies work.

Cross-region failover

Fail over the data first, then the traffic

Route 53 failover routinghealth check on the primaryus-east-1primaryload balancerapp serversAurora writerS3us-west-2recoveryload balancerapp (small)Aurora readerS3all trafficAurora Global Database: lag usually < 1 sS3 replicationonly if the check failspromote: writer here in about a minuteFail over the data first, then the traffic: promote the replica, scale up, and let DNS move the users.Data keeps flowing to the recovery region: Aurora Global Database and DynamoDB global tables replicate in about asecond, S3 replication copies objects (with RTC, 99.99% within 15 minutes). On failure, promote the secondary to writer,scale the app tier, and Route 53 answers with the recovery region once the primary’s health check fails.

Aurora Global Database replicates in about a second and promotes in about a minute; Route 53 failover moves users when a health check fails.

DNS failover timing · Going deeper

DNS failover takes detection plus the record’s TTL

0 s30 s60 s90 s120 s150 sStandard health checkevery 30 s, 3 failuresdetect 90 scached answers expire: TTL 60 s≈ 150 sFast health checkevery 10 s, 3 failuresdetect 30 scached answers expire: TTL 60 s≈ 90 sDNS failover takes detection plus the record’s TTL: here about 150 seconds, or 90 with fast checks.Route 53 health checkers probe the endpoint every 30 s (or 10 s) and call it unhealthy after 3 failures in a row. Clientsand resolvers keep using the old answer until its TTL runs out, so keep the TTL short on records that fail over.Health checks and failover answers run in Route 53’s data plane, so they keep working when a region is impaired.Do not make recovery depend on creating resources during the disaster: have them already provisioned.

Detection (30 s × 3) plus a 60 s TTL is about 150 s. Keep failover records’ TTL short.

Testing recovery

A recovery plan you have not run is a guess

0 min10 min20 min30 min40 minAlarm, page the on-call5 minAgree to fail over15 minPromote the database1 minScale up app servers12 minSwitch DNS, wait for TTL3 minCheck orders work5 minRTO target 30 mindrill took 41 minA recovery plan you have not run is a guess. This drill took 41 minutes against a 30-minute target.Rehearsing finds the slow steps. Here they were people deciding and servers starting, not the database. The fixes area pre-agreed trigger, runbooks that are scripts rather than documents, and capacity already running (warm standby).Run game days on a schedule, fail over for real, and measure RTO and RPO each time; the numbers drift as systems change.

This drill took 41 minutes against a 30-minute RTO. The slow steps were deciding and scaling, so pre-agree the trigger and keep capacity warm.

Lesson complete

You finished “Disaster recovery”

Nicely done.

Open in a new tab