Evaluate Noisy Annotators
OpenAI● MediumFULLTIMEPHONE SCREENProblem statement
Given binary ground-truth labels and one row of integer confidence scores per annotator, return each annotator's F1 score and AUROC.
The problem statement continues
ProExamples
Example 1
truth = [1,0,1]scores = [[90,20,70],[40,30,60]]return = ["1.0000,1.0000","0.6667,1.0000"]The first annotator is perfect; the second misses one positive but ranks both positives above the negative.
FastPrep Pro
Reported in 1 OpenAI interview this weekUnlock this recently reported problem
FastPrep Pro gives you full access to interview problems reported within the last week.
- Full problem statement and constraints
- 1 more worked example, explained
- Guided hints and editorial
- Run your code on real test cases
$8.25/month
$99 billed yearly — or $19 month-to-month. Cancel anytime.
Free plan — 2 of 2 free unlocks used this week