Design Petabyte-Scale Range Search for Large Rows
Serve stable bounded range queries over a year of indexed data while streaming multi-gigabyte rows outside the query service's memory.
Problem statement
For this exercise, assume an internal search service reads an already built, immutable dataset generation containing one trillion rows and five petabytes retained for one year.
Every row has one numeric range key, a unique stable row ID, bounded searchable metadata, and an immutable payload that may reach five gigabytes. An authorized client submits a half-open range over that one field and a row limit from 1 to 1,000.
The service returns matching row descriptors ordered by range key then row ID, plus an opaque continuation cursor. All pages from a cursor must read one dataset generation without skipping or duplicating rows when a newer generation appears.
The client then streams any selected payload separately, with resumable byte ranges and integrity verification; no query or streaming process may buffer a full row. Support 1,000 queries per second and return the first descriptor page within two seconds at p95.
Building or ingesting rows, arbitrary predicates, joins, aggregation, and full-text search are outside scope.
What to build
- An authorized client can query a half-open range over the indexed field and receive at most the requested number of ordered row descriptors.
- A client can continue through one immutable dataset generation without skipped or duplicated rows, or receive an explicit expired-cursor outcome.
- A client can stream an immutable selected row with byte-range resume and integrity checks without any service buffering the complete payload.
- The service bounds partition fan-out, exposes overload and partial-dependency outcomes, and reports query cost, latency, snapshot age, and stream failures.
At scale
- Assume five petabytes and one trillion indexed rows across immutable dataset generations retained for one year.
- Support 1,000 range queries per second; a request limit is between 1 and 1,000 descriptors, and popular or dense ranges may be highly skewed.
- Return the first complete descriptor page within two seconds at p95; a multi-gigabyte payload is streamed separately and has no two-second completion target.
- Rows may reach five gigabytes, so metadata, query merging, decompression, checksums, encryption, and delivery must all use bounded memory and resumable chunks.