Implement Multi-Head Attention
Problem statement
Implement a projection-free multi-head scaled dot-product attention forward pass. The inputs query, key and value are already projected row-major integer matrices. Split every row into heads contiguous slices of width w = d / heads.
For each query row and each head:
- Take the dot product of its head slice with each key row's matching slice.
- Divide each score by
sqrt(w). - Apply softmax across key rows:
p[j] = exp(score[j]) / sum(exp(score)). Subtract the maximum score before exponentiation for numerical stability. - Use those probabilities to take a weighted sum of the matching value slices.
Concatenate the head outputs in head order and return a matrix with one row per query and width d. No learned projections, masks, biases, dropout, backward pass or output projection are part of this exercise. Floating-point answers use a relative-or-absolute tolerance of 10^-6.
Function
multiHeadAttention(query: int[][], key: int[][], value: int[][], heads: int) → double[][]Examples
Example 1
query = [[1]]key = [[0],[1]]value = [[10],[20]]heads = 1return = [[17.31058578630005]]The scaled scores are 0 and 1. Their softmax weights are 1/(1+e) and e/(1+e), giving approximately 17.310586.
Example 2
query = [[1,0]]key = [[1,0],[0,1]]value = [[4,8],[6,2]]heads = 2return = [[4.53788284273999,5.0]]The first head uses scores 1 and 0, giving approximately 4.537883. The second head has equal scores, so its output is the average of 8 and 2: 5.
Constraints
- 1 <= query.length, key.length, value.length <= 20
- Each matrix row contains 1 to 12 entries.
- -20 <= every matrix entry <= 20
- 1 <= heads <= 12
- Every row in all three matrices has the same width d, and heads divides d.
- key.length == value.length