Tensor-Parallel MLP Gradients
Updated
Problem statement
A two-layer MLP computes hPre = A * x, h = ReLU(hPre), and y = B * h. Both weight matrices are row-sharded equally across numShards. A forward matrix-vector multiplication gathers every shard's output rows. During backpropagation, each shard computes its local contribution to an input gradient, and those contributions must be summed across shards.
Given the complete matrices in shard order, input vector x, upstream gradient yGrad, and shard count, return the correct weight gradients. The result is a ragged matrix containing every row of dA followed by every row of dB.
Use the standard derivatives:
ReLU'(z) = 1whenz > 0, otherwise0.- For
out = W * v,dWis the outer product ofdOutandv. dv = W^T * dOut; with row sharding this is the sum of every shard's local partial vector.
Apply the second-layer backward pass to the post-ReLU activation h, then apply the ReLU gradient using the pre-activation hPre.
Function
shardedMlpGradients(A: double[][], B: double[][], x: double[], yGrad: double[], numShards: int) → double[][]Examples
Example 1
A = [[1,2],[-1,1]]B = [[2,3],[4,-1]]x = [1,2]yGrad = [1,2]numShards = 2return = [[10,20],[1,2],[5,1],[10,2]]hPre=[5,1], so both ReLU masks are one. The gathered second-layer input gradient is B^T*yGrad=[10,1]. Thus dA=[[10,20],[1,2]], while dB=outer([1,2],[5,1])=[[5,1],[10,2]].
Example 2
A = [[1,-2],[3,-1]]B = [[1,4]]x = [1,1]yGrad = [3]numShards = 1return = [[0,0],[12,12],[0,6]]hPre=[-1,2], so the first hidden gradient is masked out. The post-ReLU activation is [0,2], giving dB=[0,6].
Constraints
1 <= A.length, B.length <= 2001 <= A[0].length <= 200A.length = B[0].length,x.length = A[0].length, andyGrad.length = B.length.- Both matrix row counts are divisible by
numShards. - All rows in each matrix have equal length.
- All values are finite and have absolute value at most
100.
More OpenAI Onsite Interview questions
- Sparse Plant Infection SimulationPHONE SCREEN · ONSITE INTERVIEW · Seen Nov 2026
- Rectangle Point Coverage CountsPHONE SCREEN · ONSITE INTERVIEW · Seen Nov 2026
- Message Event AggregationPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
- 15-Minute Chat Session TrackerPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
- Segmented Durable Key-Value StorePHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
- Durable Work Queue OperationsPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
- Versioned Social NetworkPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
- Budgeted API Inference with Out-of-Order FeedbackONSITE INTERVIEW · Seen Oct 2026
- Plant Infection Simulation, Part 4: Death CountdownPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
- Count Friend CirclesPHONE SCREEN · ONSITE INTERVIEW · Seen Oct 2026
Similar questions at other companies
- Transformer Attention and SwiGLU Forward PassGoogle · ONSITE INTERVIEWGoogle interview questions
- Scaled Dot-Product AttentionAmazon · PHONE SCREEN
- Vanilla RNN Hidden StatesNVIDIA · PHONE SCREENNVIDIA interview questions
- Deterministic K-Means AssignmentsMeta · PHONE SCREEN · ONSITE INTERVIEWMeta interview questions
- Cache Shared Image Pipeline PrefixesAnthropic · ONSITE INTERVIEWAnthropic interview questions
- Cats, Rabbits, and Snails Board GameAsana · ONSITE INTERVIEW
- Deterministic K-Means AssignmentsMicrosoft · PHONE SCREEN · ONSITE INTERVIEWMicrosoft interview questions
- Vectorized K-Means AssignmentsSambaNova Systems · ONSITE INTERVIEWSambaNova Systems interview questions
- Matrix MultiplicationeBay · ONSITE INTERVIEWeBay interview questions
- Spiral MatrixNike, Inc. · ONSITE INTERVIEWNike, Inc. interview questions