Design Short- and Long-Term Member Churn Prediction
Predict, for every recently active member, the chance of no activity in the next 2 days and in the next 28 days, keep the scores fresh, serve them to re-engagement systems, and evaluate and launch models safely.
Problem statement
Design the machine learning system that predicts which members of a social product are about to stop using it, over a short and a long horizon, and plan how it is evaluated and served.
For this exercise, assume a professional social network where members read the feed, search jobs, send messages and post. A member is active on a day if they open the app or the website while signed in.
Using each member's history up to the moment of scoring, the system produces two probabilities for every scored member: - short-term churn: the member has no active day in the next 2 days; - long-term churn: the member has no active day in the next 28 days.
For this exercise, also assume: - Scored members are those active at least once in the last 28 days; anyone inactive for longer already counts as churned and is not scored.
- The scores are read by a re-engagement service, which compares each probability with thresholds it sets to decide whether to send a reminder or a digest, and by a growth dashboard. Choosing and sending messages is out of scope.
- Activity is lower on weekends and around public holidays, and holidays differ by country. - A score must never use data that a member has deleted, and members who opted out of personalized processing get a score built only from non-personal signals such as their country and tenure.
Building the event logging pipeline and the messaging system are out of scope.
What to build
- Every scored member has a short-term (next 2 days) and a long-term (next 28 days) churn probability, refreshed at least once a day.
- The re-engagement service can read a member's current scores, with the time they were computed and the model version, for one member at a time.
- When a member becomes active on a day, within 1 hour their short-term score is replaced by a prediction for the next 2 days after that day, made with that day's activity.
- A new model's quality is known for both horizons, by country and member segment, before it replaces the current one, and a bad model can be rolled back.
At scale
- For this exercise, assume about 400 million members are scored each day, about 120 million are active on a typical weekday, and members produce about 30 billion activity events a day.
- For this exercise, assume the daily refresh must score every member within 4 hours, and the re-engagement service reads up to 20,000 scores a second at peak with a p99 of 50 ms.
- For this exercise, assume about 55% of scored members have no active day in the next 2 days on a typical weekday (more before weekends and holidays) and about 10% have none in the next 28 days.
- For this exercise, assume a 28-day label is known only 28 days after the prediction date, and the event store keeps 13 months of history.