Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Smile-Detection Evaluation Platform

Verify a photo smile detector with versioned ground truth, slice-aware metrics, reproducible comparisons, and guarded rollout evidence.

Problem statement

Design the system used to verify that a photo feature correctly detects smiling faces. For this exercise, assume the detector returns face boxes, smile scores, and a pinned model version for an authorized photo reference.

Reviewers create versioned ground truth by matching detected and labeled faces and recording smile labels plus adjudication.

The platform runs reproducible offline evaluations, reports detection and smile-classification metrics across declared non-sensitive quality slices such as lighting, blur, occlusion, face size, and camera-client version, compares candidate and baseline versions, and preserves examples behind access controls.

A guarded rollout records production predictions and delayed reviewed outcomes so drift and regressions can be detected without treating unlabeled traffic as truth.

Define matching and threshold semantics before computing metrics, expose uncertainty and insufficient sample sizes, and never overwrite historical results when labels or metrics change.

Design data contracts, labeling and adjudication, dataset publication, evaluation execution, metric storage, comparison, rollout monitoring, privacy, and recovery. Building the detector model, photo hosting, face recognition, identity inference, demographic inference, and end-user photo editing are outside scope.

What to build

  1. Create reviewed immutable dataset versions containing face boxes, smile labels, and adjudication evidence.
  2. Run one detector version reproducibly and compute declared detection and smile metrics under pinned semantics.
  3. Compare candidate and baseline results by non-sensitive image-quality and client-version slices with confidence and coverage metadata.
  4. Record privacy-safe production predictions and reviewed outcomes to detect drift and trigger a guarded rollback.

At scale

  • For this exercise, assume 50 million access-controlled evaluation photos, 10 million new candidate cases per week, and up to 100 concurrent evaluation runs.
  • For this exercise, a standard one-million-photo evaluation should finish within two hours and publish a queryable result within five minutes after completion.
  • For this exercise, retain immutable dataset and evaluation metadata for two years while source photos follow their owning privacy and deletion policy.
  • For this exercise, evaluator retries, worker failure, and duplicate prediction events must not count a photo, face, or metric contribution twice.