Diagnose and Scale a Multi-Region Cloud Service
Use load-balancer, application, and database evidence to evolve a regional monolith into a safe multi-region service with versioned infrastructure and reversible migrations.
Problem statement
For this exercise, assume you inherit a versioned record API running behind one regional load balancer. Stateless replicas of one monolith serve idempotent create and update, point reads, and stable newest-first tenant lists from a relational primary; background workers handle secondary effects.
Traffic and stored data will grow sharply while the service expands to three regions. You must first show how load-balancer access logs, application metrics and traces, slow-query evidence, database load, and replica lag identify the limiting boundary; do not begin by replacing the system with a final architecture.
Then evolve compute, caching, read replicas, data placement, and regional routing only when measurements justify them.
Use versioned infrastructure as code to create and change regional cells, and show how one independently pressured capability can be extracted from the monolith without dual ownership or an unobservable cutover.
An acknowledged write must survive an ordinary instance or availability-zone failure; a total home-region failure may lose at most 30 seconds of acknowledged writes and must recover service within 15 minutes. Retries for one immutable mutation must converge on one result.
A writer's point read must observe its successful version, while tenant lists may lag by up to five seconds when they expose a stable cursor and as-of position.
Design the data and operational model, client and control contracts, diagnostic telemetry, staged architecture, overload behavior, regional recovery, deployment and infrastructure workflow, and migration plan. Full-text search, analytics, binary-object storage, and active-active multi-region writes are outside scope.
What to build
- Clients can create, update, point-read, and page tenant records through an available regional route while the service preserves the declared write and read-freshness contract.
- Operators can correlate load-balancer, service, dependency, query, database, and replica evidence to identify which boundary causes a latency, error, or capacity symptom.
- The platform adds compute, cache, replica, data, or regional capacity while preserving acknowledged writes during ordinary instance or availability-zone failures and bounding total-region recovery by explicit readiness, lag, RTO, and RPO rules.
- Teams can apply versioned infrastructure and application changes and extract one monolith capability through observable, validated, and reversible stages with one authoritative writer.
At scale
- For this exercise, grow from 3,000 requests per second to 90,000 peak requests per second, with 85% reads, threefold tenant bursts, and 120 million records occupying 12 TB before indexes and replicas.
- For this exercise, operate in three regions with 99.95% monthly availability, at most 250 milliseconds p95 for healthy same-region reads, and at most 500 milliseconds p95 for writes.
- For this exercise, preserve read-your-write behavior for point reads, permit at most five seconds of staleness for declared list reads, and lose no acknowledged home-region write during an ordinary instance or availability-zone failure.
- For this exercise, recover regional service within 15 minutes with at most 30 seconds of acknowledged-write loss in a total home-region failure, keep enough failover headroom to absorb one region's critical traffic, and roll out infrastructure, releases, and data ownership through bounded canaries with explicit validation and rollback gates.