Design an Integrity-First File Archive
Keep files for years so that every retrieval returns exactly the bytes that were archived, silent corruption is found and repaired, and nothing is deleted before its retention ends.
Problem statement
Design a file archive whose first priority is data integrity.
For this exercise, assume the archive is an internal service. Other teams send it files they must keep for years, such as reports and records kept for audits. A file is written once and never changed. It is read rarely, mostly when someone needs to restore it or show it to an auditor.
Each file has a retention period chosen by the team that archived it, and a file may also be put on legal hold. Until its retention ends and while it is on hold, nobody can delete or change it, administrators included. After retention ends, the owning team may delete it.
Integrity means three things here. A retrieval returns exactly the bytes that were archived, or an error, never wrong bytes. Damage that happens while a file sits in storage, such as a failing disk, flipped bits or a software bug, is found and repaired before the file is lost.
An auditor can check that a given file has not changed since the day it was archived.
Search over file contents, versioning, sharing outside the company and the clients' own systems are out of scope. You may use any technologies you like.
What to build
- A client archives a file with its own checksum and a retention period, and gets an archive ID only after the file is stored and verified.
- A client retrieves a file by archive ID and receives exactly the archived bytes, or an error if they cannot be verified.
- A file cannot be deleted or changed before its retention ends or while it is on legal hold; after that, its owner may delete it.
- The archive regularly checks every stored copy, repairs damaged copies from healthy ones, and lets an auditor confirm that a file is unchanged since it was archived.
At scale
- For this exercise, assume about 5 million files are archived a day, averaging about 1 MB and up to 50 GB each, which is about 5 TB a day; the archive will hold about 10 PB and about 10 billion files within a few years.
- For this exercise, assume retrievals are rare, a few thousand a day, and a file of 1 GB or less should start arriving within 5 minutes.
- For this exercise, assume the archive runs in three data centers and must keep every file readable after losing any one of them, and every stored copy must be checked for damage at least once every 90 days.