Behavioral questions

Yahoo

Close to launch, your model beats the baseline in offline tests, but an online experiment or a check by another team on live traffic shows the product's key metric dropping. As the senior ML engineer, what steps do you take, and how do you tell the people involved?

BehavioralLast seen Nov 16, 2026 · Machine learning engineer
OwnershipAmbiguity

What they're looking for

This scenario tests judgment under launch pressure. The interviewer wants to see that you do not ship a model that hurts the product metric just because it won offline, that you look for the cause in a structured way (data shift, label gaps, logging bugs, metric mismatch), and that you communicate early and plainly with product and partner teams. A senior answer includes a clear decision path and a plan to close the gap.

What works

  • Say first that you would pause or limit the launch while you investigate, and why.
  • List the likely causes you would check, such as training data that differs from live traffic or a bug in logging.
  • Explain how you would tell product and the partner team what you know, what you do not, and when you will know more.
  • Describe the decision rule for launching later, and any safe partial launch.

What hurts

  • Arguing that the offline result proves the online test is wrong.
  • Quietly tuning the model until the online number looks good without finding the cause.
  • Waiting until launch day to tell product about the problem.

Sample answer

Think of your own story first, then compare.