Design Context Window Management for an AI Assistant
Keep a long-running assistant grounded when durable conversation history no longer fits in one model request.
Problem statement
Design the context-management layer for a multi-turn AI assistant after its accumulated conversation exceeds the model's context limit.
For this exercise, assume the model accepts at most 128,000 tokens and the service reserves 28,000 tokens for system instructions, retrieved evidence, and the response.
For this exercise, assume every user and assistant message is durably retained, while each model call receives at most 100,000 tokens of selected conversation context.
For this exercise, assume the assistant handles 2,000 prompt submissions per second across 200,000 active conversations and must admit a prompt within 300 ms at p99 before generation begins.
For this exercise, assume exact transcript export is required, but model training, voice input, and cross-user memory are out of scope.
What to build
- Retain the complete ordered transcript for reopening and export even when older messages are excluded from a model call.
- Build a context package that never exceeds the request budget and includes system instructions, recent turns, durable summaries, and relevant older evidence.
- Track which messages or summaries support each selected context item so stale or corrected facts can be invalidated.
- Give concurrent submissions on one conversation an explicit ordering and prevent a response from silently using the wrong history version.
At scale
- For this exercise, assume 2,000 prompt submissions per second across 200,000 active conversations.
- For this exercise, target p99 prompt admission and context-plan creation under 300 ms, excluding model generation.
- For this exercise, a model request has a 128,000-token hard limit and no more than 100,000 tokens may hold selected conversation context.
- For this exercise, preserve the exact transcript for export even after summaries are regenerated.