Large Structured Data Filesystem
Design a distributed filesystem for large structured datasets with reliable block writes and reads.
Problem statement
For this exercise, assume the source-backed system identity and the explicitly bounded practice contract that follow. Design an HDFS-like distributed filesystem for large structured-data files.
Clients create directories, create a file, stream ordered blocks to storage workers, close the file so readers can see it, list paths, inspect file metadata, and read a closed file.
File size, ingest rate, replication factor, region layout, and read pattern are intentionally open interview inputs; state bounded assumptions and separate namespace authority from bulk data transfer.
What to build
- Create directories and path-addressed files under one authoritative namespace.
- Stream ordered file blocks to storage workers and expose the file only after a successful close.
- List directories and inspect file length, block locations, ownership, and closed state without scanning bulk bytes.
- Read a closed file from replicated blocks and detect a missing or corrupt replica.
At scaleExercise boundary
- File size, file count, ingest rate, replication factor, region layout, and scan pattern are intentionally unspecified; choose bounded assumptions.
- File bytes can be much larger than namespace metadata and must not pass through the namespace authority.
- Readers must never observe an open file as a successfully closed complete file.