Design Data-Loss Prevention and Recovery for a Distributed Store
Protect acknowledged records through replication, backups, corruption detection, and rehearsed recovery.
Problem statement
Design the durability and recovery plan for a distributed record service.
For this exercise, assume an acknowledged write has a zero-data-loss objective within one region, a regional disaster may use an asynchronously replicated recovery copy with an explicit bounded recovery point, and accidental deletion or silent corruption must be recoverable from immutable backups and a change log.
Define acknowledgement, replication, backup, restore, verification, failover, reconciliation, and operator controls. Business query features and a particular database vendor are outside scope.
What to build
- Acknowledge a record only after the declared regional durability condition is met.
- Detect missing, divergent, or corrupt replicas and preserve evidence before repair.
- Restore a selected point in time from immutable material and validate it before promotion.
- Fail over with explicit RPO and fencing, then reconcile without split-brain writes.
At scale
- For this exercise, assume 200,000 writes per second, 500 TB of retained data, and a seven-year compliance subset.
- For this exercise, acknowledged writes target zero data loss for a single-zone failure; a full-region disaster has a documented RPO of five minutes.
- For this exercise, recover the critical serving subset within 30 minutes and the complete dataset within 12 hours.
- For this exercise, backups are immutable, encrypted, cross-account, and restored in a scheduled verification exercise rather than trusted by existence alone.