Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Large-Scale File Change Watcher

Detect changes across millions of files, publish a durable Kafka-backed event stream, and expose and reconcile detection gaps without inventing completeness.

Problem statement

For this exercise, assume teams register directory roots and path filters on a distributed file estate, then consume create, modify, delete, and rename notifications through a Kafka-compatible durable log. A watcher may receive native filesystem notifications, periodic metadata scans, or both.

Consumers need a resumable cursor and may process events more than once; the service must preserve per-file causal order where practical, identify duplicates, and expose when a watch is catching up or requires reconciliation.

A restart, notification-buffer overflow, unavailable host, or Kafka outage must not silently create an unreported gap. File contents, malware scanning, end-user file synchronization, and cross-file transactional snapshots are outside scope.

What to build

  1. For this exercise, assume a team can create, pause, update, and inspect a versioned watch over directory roots and path filters.
  2. For this exercise, assume the service can detect create, modify, delete, and rename changes and assign each accepted observation a stable identity.
  3. For this exercise, assume authorized consumers can read a durable Kafka-backed stream and resume from a committed cursor with documented ordering and duplicate semantics.
  4. For this exercise, assume operators and consumers can observe detection gaps, trigger bounded reconciliation, and distinguish live events from recovered changes.

At scale

  • For this exercise, assume 100 million tracked paths across 10,000 hosts and a sustained 200,000 file changes per second with ten-times bursts.
  • For this exercise, 99% of healthy-host changes should enter the durable log within five seconds, while a full reconciliation may take hours.
  • For this exercise, retain the change log for seven days and watch metadata plus reconciliation checkpoints for 30 days.
  • For this exercise, hosts, watcher agents, metadata stores, and Kafka may fail independently; delivery is at least once and consumers must never be promised global ordering.