Design a Scalable Artifact Repository
Design a secure, reliable service for publishing immutable build artifacts and fetching them at large scale.
Problem statement
For this exercise, assume engineering teams publish versioned binary artifacts and later fetch exact versions for builds and deployments. A normal download must never serve an artifact that is incomplete, corrupted, quarantined, or rejected by security policy.
Design the repository's upload, validation, storage, and download paths. Explain how the system scales, how operators measure reliability, how new artifact formats and scanners can be added, and how malicious uploads are isolated. Dependency resolution, build execution, and malware remediation are out of scope.
What to build
- An authenticated publisher can upload an artifact under a namespace, name, and version, then receive a durable accepted, quarantined, or rejected outcome.
- An authorized client can resolve and download an exact approved version with end-to-end integrity verification.
- The repository scans completed uploads and prevents quarantined or rejected artifacts from entering the normal download path.
- Operators can add artifact formats, metadata extractors, and security scanners without redesigning the core storage path.
At scale
- Assume 20,000 completed publishes and 20 million downloads per day, with download traffic substantially burstier than uploads.
- Artifacts range from a few kilobytes to 5 GB, and the repository retains about 5 PB of immutable blob data.
- Target 99.99% availability for approved downloads and a p95 below 200 ms to authorize and begin a cacheable download, excluding transfer time.
- Assume asynchronous scanning may take up to 15 minutes; metadata has a 15-minute recovery-point objective and a one-hour recovery-time objective.