Design Comment Ranking for Discussion Threads
Rank the comments under a post so readers see the most valuable discussion first, keep the order fresh as votes arrive, and train the ranker from biased engagement data.
Problem statement
Design the system that decides the default order of comments under a post on a large discussion site, using machine learning.
For this exercise, assume comments form a tree: each comment is either top-level or a reply to another comment. When a reader opens a post, they see a first page of top-level comments, each with a few of its highest-ranked replies, and they can load more comments or expand a reply branch.
The default order is the same for every reader of a thread at a given moment, so it can be shared and cached; per-reader personalization is out of scope. Readers can upvote, downvote, reply, collapse a branch and report a comment.
Moderators and authors can remove or delete comments, and a removed or deleted comment must never be shown in the ranked list.
The goal is to put the comments readers find most valuable near the top, while a new comment still gets a fair chance to be seen and a burst of coordinated votes cannot easily push a comment to the top.
Design how engagement is logged, how training labels and features are built, how the ranker is trained and evaluated, and how the ranked order is computed, kept fresh and served. Comment storage, the posting and voting user interface, spam classification of comment text, and notifications are outside scope.
All counts and deadlines below are practice assumptions.
What to build
- A reader opening a post gets a page of top-level comments in the default ranked order, each with its top-ranked replies, and can load more comments or expand a branch without seeing a comment twice or missing one.
- Impressions with their position and reader actions (upvote, downvote, reply, collapse, report) are recorded once each and attributed to the ranking version that produced the order the reader saw.
- New votes, new comments and removals change the ranked order within the freshness targets, and a removed or deleted comment is never returned.
- Operators build labeled training data from logged engagement, compare a candidate ranker with the current one offline and online, and release or roll back a ranker version.
At scale
- For this exercise, assume a single viral thread can receive 1,000 votes per second and hold 100,000 comments, while most active threads have a few hundred comments.
- For this exercise, assume a peak of 20,000 thread opens per second, and the first page of 50 top-level comments must be served within 200 ms at the 95th percentile.
- For this exercise, assume a vote should be reflected in the ranked order within 2 minutes, a new comment should appear in some readers' first pages within 10 minutes when its thread is active, and a removal must stop the comment from being returned within 30 seconds.
- For this exercise, assume a ranker is retrained daily from the last 30 days of logged impressions, and a new version is released only after an online comparison against the current one.