Design Real-Time Chatbot Training and Evaluation
Serve conversational answers while building a versioned, auditable data and reinforcement-learning evaluation loop.
Problem statement
Design a chatbot that answers user questions in real time and supports an iterative training and evaluation loop. The reported interview explicitly asks how to collect training data, define reinforcement-learning tasks, identify weaknesses in reward design, and mitigate them.
For this exercise, assume authenticated users hold bounded multi-turn sessions; each answer is generated under one pinned model, prompt, retrieval-policy, and reward-definition version.
A versioned reinforcement-learning task defines its observation or input, permitted action or output, episode or trajectory boundary, terminal outcomes, and linked reward definition.
The serving path records only consented or otherwise policy-authorized, purpose-limited, privacy-filtered turn evidence and explicit user feedback without blocking the response.
Authorized reviewers may accept evidence into immutable dataset versions; deletion or consent revocation prevents that evidence from entering successor datasets and revokes candidate example access without rewriting historical audit receipts.
Evaluation runs compare candidate policies against a fixed baseline, and promotion requires quality and safety thresholds across declared slices. A retry must not duplicate a turn, feedback item, training example, trajectory, or evaluation case.
Design the online serving path, evidence eligibility and authorization contracts, labeling and dataset publication, reinforcement-learning task and reward lifecycle, offline evaluation, safe promotion, rollback, and audit.
General web crawling, training foundation models from scratch, moderation-policy authoring, and building the underlying model runtime are outside scope.
What to build
- Serve ordered, retry-safe conversational turns under pinned model and policy versions.
- Capture only authorized, purpose-limited, privacy-filtered turn, outcome, and reviewed feedback evidence with stable identities and revocation state.
- Version each reinforcement-learning task's observation, action, trajectory boundary, terminal outcomes, and linked reward definition.
- Build immutable dataset and reward-definition versions and compare candidate policies reproducibly.
- Promote or roll back one reviewed serving version while preserving audit and slice-level quality evidence.
At scale
- For this exercise, assume 50,000 concurrent sessions and 5,000 new turns per second at peak.
- For this exercise, healthy requests should begin streaming within one second at p95 and complete within ten seconds at p95.
- For this exercise, retain privacy-filtered turn evidence for 90 days and immutable reviewed dataset and evaluation metadata for one year.
- For this exercise, accepted turns and feedback must survive a process or single-zone failure, while model unavailability returns an explicit degraded outcome.