FastPrepTensor-Parallel MLP Gradients
Problems/Tensor-Parallel MLP Gradients

Tensor-Parallel MLP Gradients

OpenAI logoOpenAI● HardFULLTIMEONSITE INTERVIEW

Updated Nov 2026

QuestionFunctionExamplesConstraints

Problem statement

Source note: This practice problem is backed by a real OpenAI onsite debugging round. The source supplied pure NumPy tensor-parallel forward and backward functions, a blocking all_gather primitive, and five concrete gradient bugs. The language-neutral matrix interface below preserves the same row-sharded computation while making it executable in Java, Python, and C++.

A two-layer MLP computes hPre = A * x, h = ReLU(hPre), and y = B * h. Both weight matrices are row-sharded equally across numShards. A forward matrix-vector multiplication gathers every shard's output rows. During backpropagation, each shard computes its local contribution to an input gradient, and those contributions must be summed across shards.

Given the complete matrices in shard order, input vector x, upstream gradient yGrad, and shard count, return the correct weight gradients. The result is a ragged matrix containing every row of dA followed by every row of dB.

Use the standard derivatives:

  • ReLU'(z) = 1 when z > 0, otherwise 0.
  • For out = W * v, dW is the outer product of dOut and v.
  • dv = W^T * dOut; with row sharding this is the sum of every shard's local partial vector.

Apply the second-layer backward pass to the post-ReLU activation h, then apply the ReLU gradient using the pre-activation hPre.

Function

shardedMlpGradients(A: double[][], B: double[][], x: double[], yGrad: double[], numShards: int) → double[][]

Examples

Example 1

A = [[1,2],[-1,1]]B = [[2,3],[4,-1]]x = [1,2]yGrad = [1,2]numShards = 2return = [[10,20],[1,2],[5,1],[10,2]]

hPre=[5,1], so both ReLU masks are one. The gathered second-layer input gradient is B^T*yGrad=[10,1]. Thus dA=[[10,20],[1,2]], while dB=outer([1,2],[5,1])=[[5,1],[10,2]].

Example 2

A = [[1,-2],[3,-1]]B = [[1,4]]x = [1,1]yGrad = [3]numShards = 1return = [[0,0],[12,12],[0,6]]

hPre=[-1,2], so the first hidden gradient is masked out. The post-ReLU activation is [0,2], giving dB=[0,6].

Constraints

  • 1 <= A.length, B.length <= 200
  • 1 <= A[0].length <= 200
  • A.length = B[0].length, x.length = A[0].length, and yGrad.length = B.length.
  • Both matrix row counts are divisible by numShards.
  • All rows in each matrix have equal length.
  • All values are finite and have absolute value at most 100.

More OpenAI Onsite Interview questions

  • Sparse Plant Infection SimulationPHONE SCREEN · ONSITE INTERVIEW · Seen Nov 2026
  • Rectangle Point Coverage CountsPHONE SCREEN · ONSITE INTERVIEW · Seen Nov 2026
  • Message Event AggregationPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
  • 15-Minute Chat Session TrackerPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
  • Segmented Durable Key-Value StorePHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
  • Durable Work Queue OperationsPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
  • Versioned Social NetworkPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
  • Budgeted API Inference with Out-of-Order FeedbackONSITE INTERVIEW · Seen Oct 2026
  • Plant Infection Simulation, Part 4: Death CountdownPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
  • Count Friend CirclesPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026

Similar questions at other companies

  • Transformer Attention and SwiGLU Forward PassGoogle · ONSITE INTERVIEWGoogle interview questions
  • Scaled Dot-Product AttentionAmazon · PHONE SCREEN
  • Vanilla RNN Hidden StatesNVIDIA · PHONE SCREENNVIDIA interview questions
  • Deterministic K-Means AssignmentsMeta · PHONE SCREEN · ONSITE INTERVIEWMeta interview questions
  • Cache Shared Image Pipeline PrefixesAnthropic · ONSITE INTERVIEWAnthropic interview questions
  • Cats, Rabbits, and Snails Board GameAsana · ONSITE INTERVIEW
  • Deterministic K-Means AssignmentsMicrosoft · PHONE SCREEN · ONSITE INTERVIEWMicrosoft interview questions
  • Vectorized K-Means AssignmentsSambaNova Systems · ONSITE INTERVIEWSambaNova Systems interview questions
  • Matrix MultiplicationeBay · ONSITE INTERVIEWeBay interview questions
  • Spiral MatrixNike, Inc. · ONSITE INTERVIEWNike, Inc. interview questions
All OpenAI interview questionsSee OpenAI hiring insights
Saved
class Solution {
    public double[][] shardedMlpGradients(double[][] A, double[][] B, double[] x, double[] yGrad, int numShards) {
      // write your code here
    }
}
Checking account…