Design Cross-Region Versioned Map Reconciliation
Synchronize two 32 GB in-memory maps with deterministic conflict rules while minimizing cross-region transfer.
Problem statement
Two nodes, one in the USA and one in Europe, each hold an approximately 32 GB in-memory map. Every key exists on both nodes and maps to data plus an integer version; at most five percent of entries differ. Reconcile them offline so both become identical.
A higher version wins; if versions are equal but data differs, the USA value wins. Cross-region transfer is expensive, local CPU, RAM, and NVMe are plentiful, and completion within a few hours is acceptable.
A working solution is required: include executable pseudocode or code-level logic for the comparison, repair, retry, and verification loop rather than stopping at a component diagram. Minimize network bytes; multiprocessing and online concurrent writes are outside scope for this exercise.
What to build
- Identify differing entries without transferring both complete 32 GB maps.
- Apply the declared version rule and USA tie-break identically on both nodes.
- Transfer only comparison summaries and values needed to repair discrepancies.
- Resume interrupted reconciliation idempotently and prove both final maps are equal.
At scale
- The source fixes each in-memory map at approximately 32 GB and discrepancies at no more than five percent of entries.
- The source allows powerful local CPU, 128 GB RAM, fast NVMe storage, and high bandwidth, but treats cross-region data transfer as the dominant cost.
- The source permits an offline background run that completes within a few hours and excludes multiprocessing from the required phase.
- For this exercise, comparison and repair work must be resumable after either node or the network fails without retransferring already verified ranges.