Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Large-Model Inference Serving Platform

Serve versioned large models on shared accelerators with shard-aware scheduling, continuous batching, bounded admission, streaming recovery, stable experiments, and safe fallback or rollback.

Problem statement

For this exercise, assume a multi-tenant large-model inference serving platform in which clients submit a version-pinned generation request with input token count, maximum output, sampling parameters, deadline, and optional streaming response.

The gateway admits bounded work, routes it to either a compatible model replica or a complete coordinated shard group, and the engine forms continuous batches while managing accelerator memory and a prefix or key-value cache.

Assume a request has one stable identity and serving assignment across queueing, streaming, cancellation, retry, fallback decisions, and final usage accounting.

The platform distinguishes rejected, queued, running, canceled, completed, failed, and outcome-unknown states; a client reconnect resumes from a bounded output cursor or reads the authoritative result. Multiple immutable model-runtime releases may serve traffic concurrently under a stable A/B or canary assignment.

Rollout, fallback, and rollback change routing for new requests without silently changing the release or shard plan already pinned to accepted work.

Design admission, traffic scheduling, model sharding, routing, batching, memory, cache, streaming, cancellation, fairness, multi-version deployment, overload protection, recovery, isolation, and observability. Training models, prompt authoring, and application-specific moderation policy are outside scope.

What to build

  1. Accept one retry-safe version-pinned generation request with deadline and output bounds.
  2. Place admitted requests on compatible replicas or complete coordinated shard groups and continuously batch without violating fairness or deadlines.
  3. Stream attributable output and recover reconnect, cancellation, retry, and uncertain outcomes by the original request identity.
  4. Canary, A/B test, activate, fall back, and roll back immutable model-runtime releases while preserving each accepted request's serving assignment and reproducible evidence.

At scale

  • Support tens of millions of concurrent online inference requests; for the bounded capacity exercise, assume 20,000 requests per second are admitted across 2,000 accelerator replicas and 2 million generated tokens per second at peak.
  • For this exercise, target two seconds p95 time to first token and 100 milliseconds p95 inter-token gap for healthy admitted interactive requests.
  • For this exercise, cap queued input plus reserved output tokens per tenant and admit sharded work only when a complete compatible shard group can be scheduled; reject before an unbounded queue, partial shard allocation, or accelerator-memory oversubscription forms.
  • For this exercise, a process or single-zone failure may interrupt a stream but must preserve one request outcome and one usage record; reconnect recovery is bounded to five minutes.