How should I practice Tensor-Parallel MLP Gradients?

Start by reading the full statement, solve the visible examples, then compare the pattern against related OpenAI onsite interview problems in FastPrep.

Is Tensor-Parallel MLP Gradients associated with OpenAI?

FastPrep tags this as a OpenAI onsite interview coding practice problem when the problem catalog has that company and stage context.

Problem · Array

Tensor-Parallel MLP Gradients

● Hard

OpenAIFULLTIMEONSITE INTERVIEW

Problem statement

A two-layer MLP computes hPre = A * x, h = ReLU(hPre), and y = B * h. Both weight matrices are row-sharded equally across numShards. A forward matrix-vector multiplication gathers every shard's output rows. During backpropagation, each shard computes its local contribution to an input gradient, and those contributions must be summed across shards.

The problem statement continues

Pro

Examples

Example 1

A = [[1,2],[-1,1]]B = [[2,3],[4,-1]]x = [1,2]yGrad = [1,2]numShards = 2return = [[10,20],[1,2],[5,1],[10,2]]

hPre=[5,1], so both ReLU masks are one. The gathered second-layer input gradient is B^T*yGrad=[10,1]. Thus dA=[[10,20],[1,2]], while dB=outer([1,2],[5,1])=[[5,1],[10,2]].

FastPrep Pro

Reported in 1 OpenAI interview this week

Unlock this recently reported problem

FastPrep Pro gives you full access to interview problems reported within the last week.

Full problem statement and constraints
1 more worked example, explained
Guided hints and editorial
Run your code on real test cases

$9/month

Pro subscription, billed yearly — or $19 month-to-month. Cancel anytime.

Free plan — 2 of 2 free unlocks used this week