Preparing your design workspace

Loading the problem and your latest saved attempt…

Design Long-Running Trace Event Grouping

Group Kafka trace events durably until a trace is complete, then hand the complete trace to one processor.

Problem statement

Design a distributed event-processing system that consumes events from Kafka and processes all events belonging to the same trace together. Every event carries a traceId, and a trace may last from seconds to several hours.

For this exercise, assume producers can emit an explicit terminal event when available; when they cannot, the product accepts a configurable inactivity deadline and marks that completion as inferred.

The system must retain trace state outside a worker process, expose whether completion was explicit or inferred, and make downstream processing idempotent. Explain grouping, lifecycle, storage, completion, retrieval, recovery, and skew handling; the downstream trace-analysis algorithm is outside scope.

What to build

  1. Accept trace events durably and deduplicate producer or broker retries.
  2. Maintain queryable ordered state for every open trace without depending on one worker's memory.
  3. Close a trace by explicit terminal evidence or a declared inactivity policy and dispatch one idempotent processing request.
  4. Expose open, completed, inferred-complete, processing, and terminal outcomes for operations and repair.

At scale

  • For this exercise, assume one million events per second at peak, fifty million concurrently open traces, and a maximum supported trace duration of 24 hours.
  • For this exercise, explicitly completed traces should become available for processing within 30 seconds at p95; inferred completion follows the configured inactivity deadline.
  • For this exercise, accepted events and open-trace state must survive a worker or availability-zone failure, while duplicate Kafka delivery is expected.
  • For this exercise, retain raw trace events for seven days and lifecycle metadata for thirty days so operators can replay or explain a trace.