Grouped-Query Attention Forward Pass
Problem statement
Compute one batch of grouped-query scaled dot-product attention from already projected query, key and value vectors. The flat arrays use token-major, head-next, coordinate-last order:
- Query coordinate
dat tokent, headhisqueries[(t * queryHeads + h) * headSize + d]. - Key or value coordinate
dat key tokens, shared headguses index(s * kvHeads + g) * headSize + d.
Query heads form contiguous groups of size queryHeads / kvHeads. Query head h uses shared head g = floor(h / groupSize).
For each query token and head, take its dot product with every key vector from shared head g, and divide every score by sqrt(headSize). Apply softmax across all key tokens. Return their value vectors' weighted sum, using those softmax weights.
For stable softmax, subtract the maximum score before exponentiation, then divide by the sum of the exponentials. Every query may use every key position; there is no causal mask in this exercise. Return the resulting vectors in the same flat order as the queries, before any final output projection. If queryCount is zero, return an empty array.
Function
groupedQueryAttention(queries: double[], keys: double[], values: double[], queryCount: int, keyCount: int, queryHeads: int, kvHeads: int, headSize: int) → double[]Examples
Example 1
queries = [1,2,3,4]keys = [0,0]values = [2,9]queryCount = 1keyCount = 1queryHeads = 4kvHeads = 2headSize = 1return = [2.0,2.0,9.0,9.0]There is one key token, so its attention weight is one. Query heads 0,1 share value head 0, whose value is 2. Heads 2,3 share value head 1, whose value is 9.
Example 2
queries = [0,0]keys = [4,-5]values = [2,6]queryCount = 1keyCount = 2queryHeads = 2kvHeads = 1headSize = 1return = [4.0,4.0]Both query heads are zero, so their two key scores are both zero. Softmax gives weights 0.5,0.5; each output is (2 + 6) / 2 = 4.
Example 3
queries = [1]keys = [0,1]values = [0,10]queryCount = 1keyCount = 2queryHeads = 1kvHeads = 1headSize = 1return = [7.310585786300049]The two scaled scores are 0 and 1. Their softmax weights are 1 / (1 + e) and e / (1 + e). The weighted value is 10e / (1 + e), about 7.3105857863.
Constraints
0 <= queryCount <= 81 <= keyCount <= 81 <= queryHeads <= 8and1 <= kvHeads <= 8.1 <= headSize <= 8queryHeadsis divisible bykvHeads.queries.length == queryCount * queryHeads * headSizekeys.length == keyCount * kvHeads * headSizeandvalues.length == keyCount * kvHeads * headSize.keys.length == values.length- There are at most
512entries in each flat array; queries may be empty, while keys and values are nonempty. - Every array value is finite and between
-10and10, inclusive.