Deploy and Monitor a Model for One Product
Move an already-trained model through shadow and canary traffic, monitor its versioned evidence, and roll back safely when a release fails.
Problem statement
For this exercise, assume one product already has a trained model and a stable prediction API. A release operator must register an immutable model artifact, stage it on the serving fleet, compare it through shadow and canary traffic, and promote or roll it back.
Design the deployment control plane, its serving integration and monitoring. Each response must identify the exact loaded model revision, and a request must not silently mix revisions. Readiness and evaluation evidence must belong to the release revision being decided.
A missing or immature quality observation is not a passed gate. Training new models, choosing a prediction target, defining the model-specific quality metric, feature engineering and multi-product tenancy are outside scope.
The training team supplies a versioned quality-gate contract; the release system consumes its pass/fail evidence and maturity information. All workload and timing values below are authored practice assumptions. The single product has one authoritative control revision and active-rollout identity.
Promotion, rollback and route-authorization renewal must match that product state as well as the rollout generation; an old rollout cannot overwrite a newer product intent. Route authorization must expire within the rollback budget even if refresh or control-plane access fails.
A router unable to prove current authorization stops admitting requests to the affected revision instead of extending stale routing; availability may degrade during that partition. The 30-second rollback promise applies throughout a pending rollout and for 24 hours after promotion.
The prior known-good model retains sufficient warm capacity for full traffic throughout that period, including the one-day quality-evidence wait.
What to build
- Register a model artifact and validate its declared API/schema compatibility before staging that exact revision.
- Expose shadow and canary phases, then promote a release only with matching worker readiness and the required mature gate evidence.
- Report operational health and delayed model-quality observations by model revision and rollout generation.
- Within the declared rollback-eligibility period, return traffic to a known-good compatible revision without replaying an uncertain promotion or accepting stale readiness; expired routing authorization cannot admit new failed-version traffic.
At scale
- Serve 20,000 prediction requests per second with a 100-millisecond p95 response target.
- The fleet has 100 workers; a two-gigabyte model takes up to five minutes to load and warm on a worker.
- During a pending rollout and for 24 hours after promotion, stop new failed-revision traffic within 30 seconds of rollback; retain full-traffic warm capacity for the prior revision throughout that eligibility period.
- Model-quality labels may arrive one day after prediction; operational errors and latency are available promptly.