Evaluate Noisy Annotators
Problem statement
Given binary ground-truth labels and one row of integer confidence scores per annotator, return each annotator's F1 score and AUROC.
- A score of at least
50predicts label1. - F1 is
2TP / (2TP + FP + FN); return0when its denominator is zero. - AUROC compares every positive item with every negative item: a higher positive score earns
1, a tie earns0.5, and a lower score earns0. Return0if either class is absent.
Format each annotator as f1,auc with exactly four digits after the decimal point, in input row order.
For this practice, round each exact metric ratio to four decimal places using round-half-up: if the discarded part is exactly halfway, round upward. For example, 0.03125 becomes 0.0313. Determine rounding from the exact counts, without first approximating the ratio in binary floating point.
Function
evaluateAnnotators(truth: int[], scores: int[][]) → String[]Examples
Example 1
truth = [1,0,1]scores = [[90,20,70],[40,30,60]]return = ["1.0000,1.0000","0.6667,1.0000"]The first annotator is perfect; the second misses one positive but ranks both positives above the negative.
Example 2
truth = [1,0]scores = [[50,50]]return = ["0.6667,0.5000"]Both items are predicted positive, while an equal-score pair counts as one half in AUROC.
Constraints
1 <= truth.length <= 500.1 <= scores.length <= 200; every score row hastruth.lengthentries.- Truth entries are 0 or 1; scores are between 0 and 100.