High-Throughput ML Object Store
Design an object and metadata platform for ML datasets and checkpoints read concurrently by thousands of GPUs.
Problem statement
For this exercise, assume datasets contain millions of small files plus large immutable shards, and thousands of GPUs may read the same shards concurrently. Training jobs also write multi-part checkpoints that become visible atomically.
The service uses origin storage, regional caches, node-local caches, and a scalable metadata namespace. It must verify integrity, control hot-key amplification, and support snapshot-consistent dataset manifests. Exclude disk firmware and the training data format itself.
What to build
- Create an immutable manifest over versioned objects and expose it atomically after validation.
- Resolve metadata and stream hot shards through cache tiers without overwhelming origin.
- Upload checkpoint parts and atomically publish one complete checkpoint generation.
At scale
- Store 100 PB, 10 billion object names, and dataset snapshots with millions of entries.
- Serve 5 TB per second aggregate reads during synchronized training startup.
- Sustain one million metadata operations per second with p99 below 20 ms on cache hits.
- A dataset snapshot or checkpoint generation is either wholly visible or absent.