FastPrepEvaluate Noisy Annotators
Problems/Evaluate Noisy Annotators

Evaluate Noisy Annotators

OpenAI logoOpenAI● MediumFULLTIMENEW GRADPHONE SCREEN
QuestionFunctionExamplesConstraints

Problem statement

Source note: The reports name a noisy-annotator exercise; one describes an open-ended discussion of F1 and AUROC, and Python without pandas is allowed. The original notebook and full scoring rules are unavailable. This practice reconstructs metric evaluation and matches the reported task at about 50%. The binary labels, score matrix, threshold, pairwise tie credit, absent-class behavior, bounds, signature and four-decimal round-half-up output are authored practice choices.

Given binary ground-truth labels and one row of integer confidence scores per annotator, return each annotator's F1 score and AUROC.

  • A score of at least 50 predicts label 1.
  • F1 is 2TP / (2TP + FP + FN); return 0 when its denominator is zero.
  • AUROC compares every positive item with every negative item: a higher positive score earns 1, a tie earns 0.5, and a lower score earns 0. Return 0 if either class is absent.

Format each annotator as f1,auc with exactly four digits after the decimal point, in input row order.

For this practice, round each exact metric ratio to four decimal places using round-half-up: if the discarded part is exactly halfway, round upward. For example, 0.03125 becomes 0.0313. Determine rounding from the exact counts, without first approximating the ratio in binary floating point.

Function

evaluateAnnotators(truth: int[], scores: int[][]) → String[]

Examples

Example 1

truth = [1,0,1]scores = [[90,20,70],[40,30,60]]return = ["1.0000,1.0000","0.6667,1.0000"]

The first annotator is perfect; the second misses one positive but ranks both positives above the negative.

Example 2

truth = [1,0]scores = [[50,50]]return = ["0.6667,0.5000"]

Both items are predicted positive, while an equal-score pair counts as one half in AUROC.

Constraints

  • 1 <= truth.length <= 500.
  • 1 <= scores.length <= 200; every score row has truth.length entries.
  • Truth entries are 0 or 1; scores are between 0 and 100.

More OpenAI problems

  • Determine the Winner of Three-Card HandsPHONE SCREEN · Seen Nov 2026
  • Sparse Plant Infection SimulationPHONE SCREEN · ONSITE INTERVIEW · Seen Nov 2026
  • Light Source Hits: Sample, Compare, and FitPHONE SCREEN · Seen Nov 2026
All OpenAI interview questionsSee OpenAI hiring insights
Saved
class Solution {
    public String[] evaluateAnnotators(int[] truth, int[][] scores) {
      // write your code here
    }
}
Checking account…