Multi-Head Attention Block
Updated
Problem statement
Write the forward pass of a multi-head self-attention block, the attention part of a Transformer layer.
The input x holds n token vectors, one per row. Each row has d values. The weights wq, wk, wv and wo are d by d matrices. The block has heads heads, and each head works on w = d / heads columns.
Return the n by d output matrix, computed with these steps:
- Project the tokens:
Q = x * wq,K = x * wkandV = x * wv. Here*is matrix multiplication, so rowiofQis rowiofxtimeswq. - Head
huses columnsh * wthrough(h + 1) * w - 1ofQ,KandV. Head 0 uses the firstwcolumns, head 1 the nextw, and so on. - For each head and each token
i, give every tokenja score, includingiitself. The score is the dot product of the head's columns ofQ[i]andK[j], divided bysqrt(w). - Turn the
nscores of tokeniinto weights with softmax:p[j] = exp(score[j]) / (sum of exp(score) over all j). - The head's output for token
iis the sum over alljofp[j]times the head's columns ofV[j]. - Place the head outputs side by side in head order. This gives an
nbydmatrixC. ReturnC * wo.
Every token can attend to every token. There is no mask, bias, dropout, residual connection or layer norm. Scores can be large, so subtract the largest score of a row before calling exp. This gives the same weights and avoids overflow.
Function
multiHeadAttentionBlock(x: double[][], wq: double[][], wk: double[][], wv: double[][], wo: double[][], heads: int) → double[][]Examples
Example 1
x = [[1.0,0.0],[0.0,1.0]]wq = [[1.0,0.0],[0.0,1.0]]wk = [[1.0,0.0],[0.0,1.0]]wv = [[1.0,0.0],[0.0,1.0]]wo = [[1.0,0.0],[0.0,1.0]]heads = 1return = [[0.6697615493266569,0.33023845067334306],[0.33023845067334306,0.6697615493266569]]All four weight matrices are the identity, so Q, K and V all equal x. There is one head, so w = 2.
For token 0, the score with itself is 1 / sqrt(2), about 0.707, and the score with token 1 is 0. Softmax turns these into weights of about 0.670 and 0.330. Row 0 is 0.670 * [1, 0] + 0.330 * [0, 1] = [0.670, 0.330]. Multiplying by the identity wo keeps it the same.
Row 1 is the mirror image: [0.330, 0.670].
Example 2
x = [[1.0,0.0],[0.0,1.0]]wq = [[1.0,0.0],[0.0,1.0]]wk = [[1.0,0.0],[0.0,1.0]]wv = [[1.0,0.0],[0.0,1.0]]wo = [[1.0,0.0],[0.0,1.0]]heads = 2return = [[0.7310585786300049,0.5],[0.5,0.7310585786300049]]This is the same input with two heads, so each head has width w = 1 and the scores are divided by sqrt(1) = 1.
- Head 0 uses column 0. Token 0 has query
1and the keys are1and0, so its scores are1and0. The weights are about0.731and0.269, and the values in column 0 are1and0, so the output is about0.731. Token 1 has query0, so both scores are0, the weights are0.5each, and the output is0.5. - Head 1 uses column 1 and works the same way with the tokens swapped:
0.5for token 0 and about0.731for token 1.
Placing the heads side by side gives [[0.731, 0.5], [0.5, 0.731]].
Example 3
x = [[1.0,2.0]]wq = [[1.0,1.0],[0.0,1.0]]wk = [[2.0,0.0],[1.0,-1.0]]wv = [[0.5,0.0],[0.0,0.5]]wo = [[0.0,1.0],[1.0,0.0]]heads = 2return = [[1.0,0.5]]There is only one token. Each head's softmax has a single score, so its weight is 1 no matter what wq and wk are. The block just returns x * wv * wo.
x * wv = [1 * 0.5, 2 * 0.5] = [0.5, 1.0], and wo swaps the two columns, so the answer is [[1.0, 0.5]].
Constraints
1 <= n <= 16, wheren = x.length1 <= d <= 8, whered = x[0].length- Every row of
xhasdvalues. Each ofwq,wk,wvandwohasdrows ofdvalues. 1 <= heads <= d, anddis divisible byheads.-8 <= x[i][j] <= 8- Every weight is between
-2and2. - Each returned value is accepted when its absolute error is at most
10^-6 * max(1, |expected value|).
More Scale AI Phone Screen questions
- Four-Player Trick-Taking Card Game SimulationOA · PHONE SCREEN · Seen Aug 2026
- Task Processor: Dependencies and DeadlinesPHONE SCREEN · Seen Aug 2026
- Best Poker Category with a JokerPHONE SCREEN · ONSITE INTERVIEW · Seen Jun 2026
- Free Time for Every Meeting RoomPHONE SCREEN · Seen Apr 2026
- Free Windows Between Busy IntervalsOA · PHONE SCREEN · Seen Apr 2026
- Order Queries by Shortest Processing TimePHONE SCREEN · Seen Apr 2026
- Party Time by Town and Quiet Gaps by CityPHONE SCREEN · Seen Jan 2026
- Party Windows and Dead Zone TimePHONE SCREEN · Seen Oct 2025
- Overlap Duration of Two Time IntervalsPHONE SCREEN · Seen Jun 2025
- Top-K SamplerPHONE SCREEN · Seen Mar 2025
Similar questions at other companies
- Choose the Next Minesweeper ActionRunloop · PHONE SCREEN
- Best-Fit Plane Normal from 3D PointsThe Voleon Group · PHONE SCREEN
- Prefix Matrix Products and AutogradOpenAI · PHONE SCREEN · ONSITE INTERVIEWOpenAI interview questions
- Town Temperature Regression AnalysisCitadel · PHONE SCREENCitadel interview questions
- Scaled Dot-Product AttentionAmazon · PHONE SCREEN
- Implement Multi-Head AttentionApple · PHONE SCREENApple interview questions
- Range of a Third CorrelationTower Research Capital · PHONE SCREEN
- Logistic Regression DiagnosticsLinkedIn · PHONE SCREENLinkedIn interview questions
- Best Starting Cell for Restaurant DistanceDoorDash · PHONE SCREENDoorDash interview questions
- Column and Row Parallel Linear LayersNVIDIA · PHONE SCREENNVIDIA interview questions