Design a Documentation Q&A Chatbot
Design a chatbot that answers developers' questions about a cloud AI platform from its current documentation, links to its sources, handles follow-up questions, and says when it does not know.
Problem statement
Design a question-answering chatbot.
For this exercise, assume a cloud AI platform wants a chatbot on its documentation site that answers developers' questions about its products: how to set something up, how an API behaves, or what a quota or error message means. The platform's documentation is the source of truth. Answers must reflect the current documentation, link to the pages they rely on, and say plainly when the documentation does not cover a question instead of guessing; in that case the bot offers a link to human support. A developer can ask follow-up questions in the same conversation, such as "how do I do that in Python?".
For this exercise, also assume a hosted large language model is available through an API, and that its built-in knowledge of the platform is incomplete and out of date.
The bot answers only from public documentation: it cannot see a developer's own projects, billing or support tickets, and it takes no actions on their behalf. Developers may be signed in or anonymous, and anything a user types is untrusted text.
Design the path from a question to an answer, how documentation changes reach the bot, and how you would know whether its answers are good. Training a new model and building the documentation site itself are out of scope.
What to build
- A developer asks a question and receives a streamed answer in plain language with links to the documentation pages it relies on.
- Follow-up questions in the same conversation are understood in the context of the earlier questions and answers.
- When documentation pages are published, changed or removed, answers reflect the change within one hour, and removed pages are no longer cited.
- When the documentation does not support an answer, the bot says so and offers a link to support; a developer can rate any answer as helpful or not, with an optional comment.
At scale
- For this exercise, assume about 50,000 documentation pages across several products, with a few hundred pages added or changed on a typical day and bursts of a few thousand during a release.
- For this exercise, assume about 2 million questions a day, with a peak of about 100 questions per second.
- For this exercise, assume the first words of an answer should appear within 2 seconds (p95) and a complete answer within 10 seconds.
- For this exercise, assume model calls are the largest cost and are billed per token, so the amount of text sent with each question matters.