Driving Log Ingestion and Query Platform
Design petabyte-scale ingestion, indexing, and search for intermittently connected vehicle logs.
Problem statement
For this exercise, assume vehicles produce high-frequency sensor, telemetry, event, and derived-feature streams while connectivity is intermittent.
The platform must ingest resumable uploads, preserve capture-time ordering and calibration metadata, tier petabytes of immutable blobs, and let engineers search by fleet, time, geography, software version, and rare driving-event labels. Exclude live remote vehicle control and raw sensor codec design.
What to build
- Resume authenticated, integrity-checked upload of immutable vehicle log segments.
- Extract and version searchable metadata, geospatial tracks, labels, and lineage without rewriting raw logs.
- Search rare event slices and create reproducible export manifests for analysis or training.
At scale
- Ingest 5 PB per day from 100,000 intermittently connected vehicles.
- Accept resumed segment uploads without duplicate storage or missing byte ranges.
- Return selective metadata queries at p95 below 3 seconds across one year of indexed data.
- Raw logs are immutable; labels and derived features are versioned and may arrive days later.