// In depth
RPO and RTO: the two numbers that drive everything #
Disaster recovery starts with two business decisions, made in writing during onboarding: RPO (recovery point objective — how much data you can afford to lose) and RTO (recovery time objective — how long you can afford to be down). Every engineering choice follows from these numbers, which makes them cost decisions as much as risk decisions:
| Tier | RPO | RTO | What it implies | Typical fit |
|---|---|---|---|---|
| Standard | 24 hours | 24 hours | Daily snapshots, documented manual restore | Internal tools, dev environments |
| Business | 15 minutes | 4 hours | Point-in-time recovery, rehearsed runbooks | Most production systems |
| Critical | ~minutes | 1 hour | Streaming replication, automated failover, warm standby | Payments, patient-facing, revenue paths |
| Zero-ish | seconds | minutes | Multi-region active/active — expensive by design | Only where the math truly justifies it |
NoteWe will talk you out of tiers you do not need. Multi-region active/active for an internal CRM is money converted directly into complexity.
The protection layers #
| Asset | Mechanism | Frequency | Retention |
|---|---|---|---|
| Databases | Continuous WAL/binlog archiving + snapshots | Point-in-time + daily | 35 days + monthly for 12 months |
| Object storage / files | Versioning + cross-region replication | Continuous | 35 days of versions |
| Infrastructure | Terraform in Git (see Infrastructure as Code) | Every change | Forever |
| Application images | Immutable registry | Every build | 90+ days |
| Configuration & secrets | Vault with versioning + sealed backup | Every change | 90 days of versions |
Point-in-time recovery deserves emphasis: with continuous archiving, we can restore a database to any second within the window — not just to last night. When the incident is "a bad migration ran at 14:32", restoring to 14:31 beats restoring to midnight by half a day of data.
Designed for ransomware: immutability & isolation #
Modern ransomware attacks backups first — encrypt the production data, delete the recovery path, then negotiate. Our backup architecture assumes exactly this attacker:
- Separate account: backups live in a dedicated account with independent credentials. Full production compromise — even admin-level — grants zero access to backup storage.
- Immutability: object-lock/WORM storage means backups physically cannot be modified or deleted during their retention window, by anyone, including us.
- Separate region: regional failure and regional compromise are both covered by the same geography.
- Restore-path independence: recovery requires nothing from the (possibly compromised) production environment — see the IaC rebuild procedure.
Restore drills: how we know it works #
A backup is a hypothesis; a restore is the experiment. Quarterly (minimum) per managed system, we run the full procedure and measure it:
- Restore a real production backup into an isolated environment — no shortcuts, no synthetic data.
- Verify integrity: row counts, checksums, application-level smoke tests against the restored copy.
- Time every step against the RTO, and record the actual data-loss window against the RPO.
- File the results in your quarterly report — including failures, which are the most valuable possible finding in a rehearsal.
Drills find the problems that only surface under reality: the restore role whose permission was trimmed, the runbook step that assumed a tool now decommissioned, the DNS TTL that would have added 40 minutes (a real cross-region drill finding — see the changelog). Every one of these found in rehearsal is a disaster that did not happen.
Failure scenarios and their playbooks #
| Scenario | Response path | Target |
|---|---|---|
| Bad deploy / data corruption | Point-in-time restore to just before the event; replay valid writes where possible | Within RPO/RTO tier |
| Accidental deletion | Versioned-storage or PITR recovery of the affected objects only | Minutes to hours |
| Infrastructure failure (AZ) | Multi-AZ redundancy absorbs it — no recovery procedure needed | Automatic |
| Regional outage | Rebuild from IaC in alternate region + restore data + DNS cutover | Per RTO tier; rehearsed |
| Ransomware / account compromise | Clean-room rebuild from immutable backups in isolated account | Per RTO tier; assumes zero trust in production |
NoteEach scenario has a written runbook an engineer can follow under stress — recovery procedures designed for the 4 a.m. version of your team, not the well-rested one.
What protection costs #
Backup storage is cheap; recovery capability is an engineering investment that scales with your RPO/RTO tier. As a rule of thumb across our managed fleet: standard-tier protection adds low single-digit percentages to infrastructure cost, business tier roughly 5–10%, and critical tier 15–30% (warm standby capacity is real capacity). We model this during onboarding so the tier choice is made with the price tag visible — and revisited yearly, because the right answer changes as the business grows.