Design an External-Model Document Enrichment Pipeline
Design a lease-based worker platform that enriches changing documents through a rate-limited external model and supports safe replay.
Problem statement
Design a pipeline that claims documents, sends their text to an external model API in batches, and writes enrichment results back to document state. Workers may crash or receive duplicate work.
A 429 response includes Retry-After and must pause the whole worker fleet sharing that provider account and model, not just the worker that saw the response. Operators must be able to replay documents updated during the last five days.
For this exercise, assume every document has a monotonically increasing content version and enrichment may be committed only when the claimed version is still current. An atomic claim has a renewable lease, and an expired lease makes the work eligible for recovery.
Assume 20 million documents, 200 document updates per second at peak, 64 KB maximum extracted text, an external provider allowance of 100 requests per second per account, and enrichment visible within 15 minutes at p95 while the provider is healthy.
Retry-After applies to the provider-account and model pair named by the response; workers honor one monotonically extended shared not-before deadline and use jittered restart admission.
Design document and job state, claim and renewal semantics, model request and writeback contracts, batching and rate admission, the shared backoff mechanism, retry and dead-letter behavior, the five-day replay workflow, observability, and recovery.
Model training, prompt authoring, document editing, and interactive end-user inference are outside scope.
What to build
- Atomically claim one document version with a renewable lease so only one active owner processes it and abandoned work can be recovered.
- Call the external model in bounded batches and commit enrichment only if the claimed document version is still current.
- Propagate a provider Retry-After deadline to every worker sharing the affected account and model before admitting more requests.
- Let operators replay all eligible document versions updated during the last five days with durable progress and idempotent outcomes.
- Expose successful, stale, retrying, exhausted, and invalid outcomes with enough lineage to inspect and safely redrive work.
At scale
- For this exercise, assume 20 million documents and 200 document updates per second at peak.
- For this exercise, assume extracted document text is at most 64 KB and enrichment should be visible within 15 minutes at p95 while the provider is healthy.
- For this exercise, assume the external account allows 100 requests per second and may return account-and-model-scoped Retry-After deadlines.
- For this exercise, assume an operator replay may cover every eligible document version updated during the preceding five days.