Design a Large-Scale File Change Watcher
Detect changes across millions of files, publish a durable Kafka-backed event stream, and expose and reconcile detection gaps without inventing completeness.
Problem statement
For this exercise, assume teams register directory roots and path filters on a distributed file estate, then consume create, modify, delete, and rename notifications through a Kafka-compatible durable log. A watcher may receive native filesystem notifications, periodic metadata scans, or both.
Consumers need a resumable cursor and may process events more than once; the service must preserve per-file causal order where practical, identify duplicates, and expose when a watch is catching up or requires reconciliation.
A restart, notification-buffer overflow, unavailable host, or Kafka outage must not silently create an unreported gap. File contents, malware scanning, end-user file synchronization, and cross-file transactional snapshots are outside scope.
What to build
- For this exercise, assume a team can create, pause, update, and inspect a versioned watch over directory roots and path filters.
- For this exercise, assume the service can detect create, modify, delete, and rename changes and assign each accepted observation a stable identity.
- For this exercise, assume authorized consumers can read a durable Kafka-backed stream and resume from a committed cursor with documented ordering and duplicate semantics.
- For this exercise, assume operators and consumers can observe detection gaps, trigger bounded reconciliation, and distinguish live events from recovered changes.
At scale
- For this exercise, assume 100 million tracked paths across 10,000 hosts and a sustained 200,000 file changes per second with ten-times bursts.
- For this exercise, 99% of healthy-host changes should enter the durable log within five seconds, while a full reconciliation may take hours.
- For this exercise, retain the change log for seven days and watch metadata plus reconciliation checkpoints for 30 days.
- For this exercise, hosts, watcher agents, metadata stores, and Kafka may fail independently; delivery is at least once and consumers must never be promised global ordering.