Design Long-Running Trace Event Grouping
Group Kafka trace events durably until a trace is complete, then hand the complete trace to one processor.
Problem statement
Design a distributed event-processing system that consumes events from Kafka and processes all events belonging to the same trace together. Every event carries a traceId, and a trace may last from seconds to several hours.
For this exercise, assume producers can emit an explicit terminal event when available; when they cannot, the product accepts a configurable inactivity deadline and marks that completion as inferred.
The system must retain trace state outside a worker process, expose whether completion was explicit or inferred, and make downstream processing idempotent. Explain grouping, lifecycle, storage, completion, retrieval, recovery, and skew handling; the downstream trace-analysis algorithm is outside scope.
What to build
- Accept trace events durably and deduplicate producer or broker retries.
- Maintain queryable ordered state for every open trace without depending on one worker's memory.
- Close a trace by explicit terminal evidence or a declared inactivity policy and dispatch one idempotent processing request.
- Expose open, completed, inferred-complete, processing, and terminal outcomes for operations and repair.
At scale
- For this exercise, assume one million events per second at peak, fifty million concurrently open traces, and a maximum supported trace duration of 24 hours.
- For this exercise, explicitly completed traces should become available for processing within 30 seconds at p95; inferred completion follows the configured inactivity deadline.
- For this exercise, accepted events and open-trace state must survive a worker or availability-zone failure, while duplicate Kafka delivery is expected.
- For this exercise, retain raw trace events for seven days and lifecycle metadata for thirty days so operators can replay or explain a trace.