How should I practice Self-Attention and Causal Masked Self-Attention?

Start by reading the full statement, solve the visible examples, then compare the pattern against related Microsoft onsite interview problems in FastPrep.

Is Self-Attention and Causal Masked Self-Attention associated with Microsoft?

FastPrep tags this as a Microsoft onsite interview coding practice problem when the problem catalog has that company and stage context.

Self-Attention and Causal Masked Self-Attention

Microsoft● HardFULLTIMEONSITE INTERVIEW

Learn

Problem statement

Implement single-head scaled dot-product self-attention over one sequence. The rows of query, key, and value correspond to the same n sequence positions; they are already projected, so no learned weight matrices are supplied.

For query position i, score key position j by dot(query[i], key[j]) / sqrt(d), where d is the query/key width. Apply maximum-subtracted softmax to the allowed scores, then return the weighted sum of the corresponding value rows.

When causal is false, every position may attend to every key. When causal is true, query position i may attend only to positions 0..i, including itself. Return an n by v matrix. There is no batch dimension, additional attention head, bias, dropout, or backward pass. Compare each returned entry with absolute tolerance 1e-6.

Function

selfAttention(query: double[][], key: double[][], value: double[][], causal: boolean) → double[][]

Examples

Example 1

query = [[0.0],[0.0]]key = [[1.0],[2.0]]value = [[4.0],[8.0]]causal = falsereturn = [[6.0],[6.0]]

Without a mask, both zero query rows give equal weight to values 4 and 8.

Example 2

query = [[0.0],[0.0]]key = [[1.0],[2.0]]value = [[4.0],[8.0]]causal = truereturn = [[4.0],[6.0]]

The first row may attend only to position zero; the second sees both positions.

Example 3

query = [[1.0,0.0],[0.0,1.0]]key = [[1.0,0.0],[0.0,1.0]]value = [[2.0,0.0],[0.0,4.0]]causal = falsereturn = [[1.339523098653,1.320953802693],[0.660476901347,2.679046197307]]

Each query favors its matching key after scaling by the square root of two.

Constraints

1 <= n <= 20, 1 <= d, v <= 8.
query and key are rectangular n by d matrices.
value is a rectangular n by v matrix.
All entries are finite real numbers between -20 and 20, inclusive.
causal is a boolean flag selecting the mask.