Preparing your design workspace

Loading the problem and your latest saved attempt…

Design Data-Loss Prevention and Recovery for a Distributed Store

Protect acknowledged records through replication, backups, corruption detection, and rehearsed recovery.

Problem statement

Design the durability and recovery plan for a distributed record service.

For this exercise, assume an acknowledged write has a zero-data-loss objective within one region, a regional disaster may use an asynchronously replicated recovery copy with an explicit bounded recovery point, and accidental deletion or silent corruption must be recoverable from immutable backups and a change log.

Define acknowledgement, replication, backup, restore, verification, failover, reconciliation, and operator controls. Business query features and a particular database vendor are outside scope.

What to build

  1. Acknowledge a record only after the declared regional durability condition is met.
  2. Detect missing, divergent, or corrupt replicas and preserve evidence before repair.
  3. Restore a selected point in time from immutable material and validate it before promotion.
  4. Fail over with explicit RPO and fencing, then reconcile without split-brain writes.

At scale

  • For this exercise, assume 200,000 writes per second, 500 TB of retained data, and a seven-year compliance subset.
  • For this exercise, acknowledged writes target zero data loss for a single-zone failure; a full-region disaster has a documented RPO of five minutes.
  • For this exercise, recover the critical serving subset within 30 minutes and the complete dataset within 12 hours.
  • For this exercise, backups are immutable, encrypted, cross-account, and restored in a scheduled verification exercise rather than trusted by existence alone.