Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Lab Machine Fleet Monitor

Monitor a geographically distributed lab-machine fleet with timely health state, alerts, history, and partition-aware recovery.

Problem statement

The source asks for a system that monitors a fleet of laboratory machines. For this exercise, assume an agent on each machine reports identity, software and hardware inventory, heartbeat, health checks, active experiment or maintenance state, and bounded metrics.

Operators need a fleet dashboard, machine history, filtered search, alerts, and maintenance silences. The service must distinguish a machine failure from delayed telemetry or a site network partition. Remote experiment scheduling, arbitrary shell access, and firmware deployment are outside scope.

What to build

  1. Machine agents can authenticate and report sequenced heartbeats, inventory changes, health checks, and bounded metrics.
  2. Operators can search and filter current machine state and open a time-ordered health and experiment history.
  3. Operators can define alert policies, receive deduplicated incidents, acknowledge them, and apply audited maintenance silences.
  4. The service can represent site-wide telemetry loss separately from individual machine failure and reconcile late reports.

At scale

  • For this exercise, assume 250,000 machines across 200 sites, with one heartbeat every 30 seconds and up to 20 health or metric samples per heartbeat.
  • For this exercise, mark telemetry stale within two minutes and deliver a new actionable incident within one additional minute at p95.
  • For this exercise, retain raw telemetry for 30 days, downsampled trends for one year, and incident and maintenance audit history for three years.
  • For this exercise, a site partition may delay hours of reports, and reconnecting agents can replay sequenced batches without duplicating state transitions.