How should I practice Adaptive Calibration Error?

Start by reading the full statement, solve the visible examples, then compare the pattern against related OpenAI phone screen problems in FastPrep.

Is Adaptive Calibration Error associated with OpenAI?

FastPrep tags this as a OpenAI phone screen coding practice problem when the problem catalog has that company and stage context.

Adaptive Calibration Error

OpenAI● MediumNEW GRADPHONE SCREEN

Problem statement

Source note: This comes from the ML round of an OpenAI research scientist phone screen set on 10-class MNIST. The report asked for an implementation of the ACE calibration metric, which bins examples and compares accuracy with predicted probability, with no need for numpy, and for test cases. It pointed to the published ACE definition (Nixon et al., 2019).

The report gave no function signature, input format, bin count, tie rule or limits. Practice choices here: the input is a probability matrix with true labels and a bin count; bins are built for every class, each bin holds the same number of examples, so the example count must be a multiple of the bin count; equal probabilities keep index order; the score is the plain average over class and bin pairs; and the size limits are ours. The report's loss-bound and curve-smoothing questions are discussed in the editorial and are not judged.

The judged task matches the report at about 70%.

A classifier gives each of n examples a probability for each of K classes. probabilities[i][c] is the predicted probability that example i is class c, and labels[i] is the example's true class.

Return the model's Adaptive Calibration Error (ACE) with numBins bins, computed like this:

For each class c, sort the examples by probabilities[i][c] from low to high. Examples with equal probability keep index order, smaller index first.
Cut this sorted list into numBins consecutive bins of n / numBins examples each.
In each bin, the accuracy is the fraction of its examples whose label is c, and the confidence is the average of their probabilities[i][c]. The bin's gap is |accuracy - confidence|.
ACE is the average gap over all K * numBins class and bin pairs.

A well-calibrated model has an ACE near 0. Because every bin holds the same number of examples, each gap is estimated from the same amount of data.

The problem statement continues

Pro

Examples

Example 1

probabilities = [[0.9,0.1],[0.6,0.4],[0.3,0.7],[0.2,0.8]]labels = [0,0,1,0]numBins = 2return = 0.25

Class 0 sorted by probability is 0.2, 0.3 | 0.6, 0.9. The low bin has confidence 0.25 and accuracy 0.5; the high bin has confidence 0.75 and accuracy 1. Both gaps are 0.25. Class 1 sorted is 0.1, 0.4 | 0.7, 0.8, with gaps |0 - 0.25| = 0.25 and |0.5 - 0.75| = 0.25. The answer is (4 * 0.25) / (2 * 2) = 0.25.

FastPrep Pro

Reported in 1 OpenAI interview this week

Unlock this recently reported problem

FastPrep Pro gives you full access to interview problems reported within the last week.

Full problem statement and constraints
2 more worked examples, explained
Guided hints and editorial
Run your code on real test cases

$8.25/month

$99 billed yearly — or $19 month-to-month. Cancel anytime.

Free plan — 2 of 2 free unlocks used this week

See OpenAI hiring insights