Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Model-Version-Aware Derived-Data Cache

Serve cached user data derived by an LLM while safely moving readers to a newly activated model version.

Problem statement

Design the service behind a client request for cached user data when that data is generated by an LLM and a newly activated model may produce a different value. For this exercise, assume each derived value is identified by user, normalized input version, prompt version, and model version.

A model activation must never cause old output to be mislabeled as new output; the product may serve explicitly labeled stale data while regeneration is pending. Cover cache lookup, version activation, invalidation or lazy regeneration, stampede control, failures, rollout, rollback, and observability.

Training the model and evaluating answer quality are outside scope.

What to build

  1. Return a derived value with the exact input, prompt, and model versions that produced it.
  2. Activate or roll back a model version without relabeling or corrupting previously cached values.
  3. Regenerate missing or stale values with bounded duplicate work and explicit pending or stale outcomes.
  4. Expose hit, stale, regeneration, version, failure, and rollback evidence.

At scale

  • For this exercise, assume 100,000 reads per second, ten million active users, and model inference that normally takes 2 to 20 seconds.
  • For this exercise, a new model version is rolled out to cohorts over hours; cache keys and records must support overlapping old and new versions.
  • For this exercise, a regional cache loss must not lose the durable derivation identity or make an unverified value authoritative.
  • For this exercise, cap concurrent regenerations per user and model version and return an explicit stale or pending response during overload.