Design a Model Deployment Platform
Let many ML teams turn trained models into online prediction services on shared capacity, with gradual rollouts that roll back automatically and no dropped requests.
Problem statement
Design an internal platform that ML teams use to deploy trained models as online prediction services.
For this exercise, assume about 50 teams use the platform. A team registers a trained model version: the model files (from about 100 MB up to about 20 GB), the serving code packaged as a container image, and the offline evaluation results.
The team then deploys that version to an endpoint, a stable address that product services call for predictions. The platform rolls the new version out gradually, watches it, and either finishes the rollout or rolls back on its own.
All teams share one pool of CPU and GPU machines, and one team's traffic spike or bad model must not hurt other teams' endpoints. Every prediction must be traceable to the exact model version that produced it.
Training models, building features and labeling data are out of scope; models arrive already trained. You may use any technologies you like.
What to build
- A team registers an immutable model version with its files, serving image and evaluation results.
- A team deploys a version to an endpoint; the platform shifts traffic gradually and promotes or rolls back automatically, and the team can roll back by hand at any time.
- Product services call an endpoint for predictions and get a response tagged with the model version that produced it, with no dropped requests during a rollout.
- Endpoints from all teams scale on a shared pool of CPU and GPU machines, with limits that keep one team from starving the others.
At scale
- For this exercise, assume about 400 live endpoints and about 200 deployments a day.
- For this exercise, assume about 300,000 predictions a second in total, with the busiest endpoint at about 30,000 a second; each endpoint sets its own latency target, typically 100 ms at the 99th percentile.
- For this exercise, assume loading a large model onto a machine takes up to about 5 minutes. During a rollout and for 24 hours after it reaches all traffic, a rollback must move all of the endpoint's traffic back to the previous version within 2 minutes; after that window, going back is a new deployment.