FastPrepMulti-Head Attention Block
Problems/Multi-Head Attention Block

Multi-Head Attention Block

Scale AI logoScale AI● HardINTERNPHONE SCREEN

Updated Nov 2026

QuestionFunctionExamplesConstraints

Problem statement

Source note: A candidate for the Scale AI Machine Learning Research Engineering Intern role (Summer 2025) had two ML coding tasks in the first live round. One of them was implementing multi-head attention as a full block.

The report does not give shapes, weights, or an input format, and does not say whether a mask is used. Here you get one sequence of token vectors and four d by d weight matrices, and you return the output after the output projection. There is no mask, bias, or layer norm, and answers are checked with a small tolerance.

Our version is about 70% the same as the reported question.

Write the forward pass of a multi-head self-attention block, the attention part of a Transformer layer.

The input x holds n token vectors, one per row. Each row has d values. The weights wq, wk, wv and wo are d by d matrices. The block has heads heads, and each head works on w = d / heads columns.

Return the n by d output matrix, computed with these steps:

  1. Project the tokens: Q = x * wq, K = x * wk and V = x * wv. Here * is matrix multiplication, so row i of Q is row i of x times wq.
  2. Head h uses columns h * w through (h + 1) * w - 1 of Q, K and V. Head 0 uses the first w columns, head 1 the next w, and so on.
  3. For each head and each token i, give every token j a score, including i itself. The score is the dot product of the head's columns of Q[i] and K[j], divided by sqrt(w).
  4. Turn the n scores of token i into weights with softmax: p[j] = exp(score[j]) / (sum of exp(score) over all j).
  5. The head's output for token i is the sum over all j of p[j] times the head's columns of V[j].
  6. Place the head outputs side by side in head order. This gives an n by d matrix C. Return C * wo.

Every token can attend to every token. There is no mask, bias, dropout, residual connection or layer norm. Scores can be large, so subtract the largest score of a row before calling exp. This gives the same weights and avoids overflow.

Function

multiHeadAttentionBlock(x: double[][], wq: double[][], wk: double[][], wv: double[][], wo: double[][], heads: int) → double[][]

Examples

Example 1

x = [[1.0,0.0],[0.0,1.0]]wq = [[1.0,0.0],[0.0,1.0]]wk = [[1.0,0.0],[0.0,1.0]]wv = [[1.0,0.0],[0.0,1.0]]wo = [[1.0,0.0],[0.0,1.0]]heads = 1return = [[0.6697615493266569,0.33023845067334306],[0.33023845067334306,0.6697615493266569]]

All four weight matrices are the identity, so Q, K and V all equal x. There is one head, so w = 2.

For token 0, the score with itself is 1 / sqrt(2), about 0.707, and the score with token 1 is 0. Softmax turns these into weights of about 0.670 and 0.330. Row 0 is 0.670 * [1, 0] + 0.330 * [0, 1] = [0.670, 0.330]. Multiplying by the identity wo keeps it the same.

Row 1 is the mirror image: [0.330, 0.670].

Example 2

x = [[1.0,0.0],[0.0,1.0]]wq = [[1.0,0.0],[0.0,1.0]]wk = [[1.0,0.0],[0.0,1.0]]wv = [[1.0,0.0],[0.0,1.0]]wo = [[1.0,0.0],[0.0,1.0]]heads = 2return = [[0.7310585786300049,0.5],[0.5,0.7310585786300049]]

This is the same input with two heads, so each head has width w = 1 and the scores are divided by sqrt(1) = 1.

  • Head 0 uses column 0. Token 0 has query 1 and the keys are 1 and 0, so its scores are 1 and 0. The weights are about 0.731 and 0.269, and the values in column 0 are 1 and 0, so the output is about 0.731. Token 1 has query 0, so both scores are 0, the weights are 0.5 each, and the output is 0.5.
  • Head 1 uses column 1 and works the same way with the tokens swapped: 0.5 for token 0 and about 0.731 for token 1.

Placing the heads side by side gives [[0.731, 0.5], [0.5, 0.731]].

Example 3

x = [[1.0,2.0]]wq = [[1.0,1.0],[0.0,1.0]]wk = [[2.0,0.0],[1.0,-1.0]]wv = [[0.5,0.0],[0.0,0.5]]wo = [[0.0,1.0],[1.0,0.0]]heads = 2return = [[1.0,0.5]]

There is only one token. Each head's softmax has a single score, so its weight is 1 no matter what wq and wk are. The block just returns x * wv * wo.

x * wv = [1 * 0.5, 2 * 0.5] = [0.5, 1.0], and wo swaps the two columns, so the answer is [[1.0, 0.5]].

Constraints

  • 1 <= n <= 16, where n = x.length
  • 1 <= d <= 8, where d = x[0].length
  • Every row of x has d values. Each of wq, wk, wv and wo has d rows of d values.
  • 1 <= heads <= d, and d is divisible by heads.
  • -8 <= x[i][j] <= 8
  • Every weight is between -2 and 2.
  • Each returned value is accepted when its absolute error is at most 10^-6 * max(1, |expected value|).

More Scale AI Phone Screen questions

  • Four-Player Trick-Taking Card Game SimulationOA · PHONE SCREEN · Seen Aug 2026
  • Task Processor: Dependencies and DeadlinesPHONE SCREEN · Seen Aug 2026
  • Best Poker Category with a JokerPHONE SCREEN · ONSITE INTERVIEW · Seen Jun 2026
  • Free Time for Every Meeting RoomPHONE SCREEN · Seen Apr 2026
  • Free Windows Between Busy IntervalsOA · PHONE SCREEN · Seen Apr 2026
  • Order Queries by Shortest Processing TimePHONE SCREEN · Seen Apr 2026
  • Party Time by Town and Quiet Gaps by CityPHONE SCREEN · Seen Jan 2026
  • Party Windows and Dead Zone TimePHONE SCREEN · Seen Oct 2025
  • Overlap Duration of Two Time IntervalsPHONE SCREEN · Seen Jun 2025
  • Top-K SamplerPHONE SCREEN · Seen Mar 2025

Similar questions at other companies

  • Choose the Next Minesweeper ActionRunloop · PHONE SCREEN
  • Best-Fit Plane Normal from 3D PointsThe Voleon Group · PHONE SCREEN
  • Prefix Matrix Products and AutogradOpenAI · PHONE SCREEN · ONSITE INTERVIEWOpenAI interview questions
  • Town Temperature Regression AnalysisCitadel · PHONE SCREENCitadel interview questions
  • Scaled Dot-Product AttentionAmazon · PHONE SCREEN
  • Implement Multi-Head AttentionApple · PHONE SCREENApple interview questions
  • Range of a Third CorrelationTower Research Capital · PHONE SCREEN
  • Logistic Regression DiagnosticsLinkedIn · PHONE SCREENLinkedIn interview questions
  • Best Starting Cell for Restaurant DistanceDoorDash · PHONE SCREENDoorDash interview questions
  • Column and Row Parallel Linear LayersNVIDIA · PHONE SCREENNVIDIA interview questions

Interview reports that mention this question

  • Scale AI ML Research Engineering Intern Interviews: 1 Experience

Scale AI interview experiences

  • Scale AI Forward Deployed Engineer Debugging Round: 1 Experience
  • Scale AI SWE Intern Debugging Interview: 9 Experiences
  • Scale AI New Grad SWE and MLE Question Bank
  • Scale AI New Grad SWE Poker Screen and Onsite: 1 Experience
  • Scale AI New Grad SWE OA Questions: 1 Experience
All Scale AI interview questionsSee Scale AI hiring insights
Saved
class Solution {
    public double[][] multiHeadAttentionBlock(double[][] x, double[][] wq, double[][] wk, double[][] wv, double[][] wo, int heads) {
        // Write your code here.
    }
}
Checking account…