Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Priority-Aware Distributed Worker Platform

Design a shared asynchronous task platform that schedules heterogeneous work by priority and tenant while making every delayed, retried, and failed task explainable.

Problem statement

For this exercise, assume product services submit heterogeneous asynchronous units of work such as search indexing, vector generation, external-system synchronization, report generation, email processing, and data-recovery tasks that rebuild a tenant's lost or damaged records and can run for hours.

Each request declares a registered task type, tenant, priority class, immutable payload reference, and idempotency key.

The platform must durably accept work, start higher-priority tasks sooner without starving normal work, run tasks on compatible horizontally scaled workers, retry recoverable failures, and expose enough history to explain why a task is queued, delayed, retried, or terminal.

A long-running attempt can fail many hours in, and the platform must handle that without hiding it or throwing away all the work already done. Queue delivery and worker execution are at least once; business handlers must use a stable logical task identity when they need effect idempotency.

Design the APIs, task and attempt state, priority and fairness policy, queue and worker topology, leases, retry and dead-letter behavior, late failures in long-running tasks, backpressure, observability, replay, and recovery.

Dependency graphs, recurring schedules, arbitrary user code, and the implementation of each registered handler are outside scope.

What to build

  1. A service can idempotently submit one immutable registered task with tenant and priority metadata and receive a durable task identifier.
  2. The platform starts higher-priority ready work sooner while preserving tenant isolation and bounded progress for normal-priority work.
  3. Compatible workers execute under fenced leases, report outcomes, and recover retryable work after duplicate delivery or infrastructure failure.
  4. Clients and operators can inspect a task's decision and attempt history, diagnose delay or failure, and explicitly replay eligible terminal work.

At scale

  • For this exercise, accept 50,000 tasks per second at peak, tolerate a five-million-task burst backlog, and support task durations from 100 milliseconds to 30 minutes for most task types; a single data-recovery attempt may run for up to 12 hours, with at most 500 such attempts running at once.
  • For this exercise, start 99% of admitted urgent tasks within five seconds and 95% of normal tasks within one minute under provisioned capacity; bulk work may wait longer but must not starve indefinitely.
  • For this exercise, support 20,000 tenants and 200 registered task types; one tenant or task type may temporarily produce 40% of submissions.
  • For this exercise, retain terminal task and attempt history for 30 days and recover accepted work after a worker, queue partition, or availability-zone failure without losing the reason for any state transition.