Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Durable Agent Workflow Runtime

Design crash-resilient agent runs with isolated tools, approval pauses, bounded branches and loops, live observation, and quota-aware fan-out.

Problem statement

For this exercise, assume authenticated tenants start long-running agent runs from an immutable workflow version and input. A run may invoke registered tools inside isolated sandboxes, pause for a human approval, choose a conditional branch, execute a bounded loop, fan out child steps, and join their terminal outcomes.

Every run, step, tool attempt, approval request, and progress event has a stable identity. Accepted runs survive coordinator, worker, sandbox, and stream-gateway restarts.

Tool execution is at least once, so each attempt receives an idempotency key and side-effecting tools must report committed, rejected, or uncertain outcomes. A reconnecting observer resumes ordered progress from a cursor.

Per-tenant quotas bound concurrent sandboxes, fan-out, loop iterations, and retained events, and one noisy tenant cannot starve others. Assume 10,000 active tenants, 20 run starts per second, 200,000 concurrent runs, up to 1,000 steps and 100,000 progress events per run, and runs lasting up to seven days.

Model internals, tool business logic, and unbounded self-modifying workflows are outside scope.

What to build

  1. A tenant can idempotently start a run from an immutable workflow version and inspect its durable state.
  2. The runtime executes registered tools in isolated sandboxes with stable attempt identities, bounded retries, and explicit uncertain outcomes.
  3. A run can pause for one versioned human approval and resume exactly once into an approved or rejected path.
  4. The runtime durably evaluates bounded branches, loops, fan-out, and joins while observers resume ordered progress after disconnect or process failure.

At scale

  • 10,000 active tenants and 200,000 concurrent runs are authored practice assumptions.
  • The runtime accepts 20 run starts per second; a run may last seven days and contain up to 1,000 steps and 100,000 progress events.
  • Per-tenant quotas bound concurrent sandboxes, fan-out, loop iterations, retained events, and retry work.