Implement multi-head self-attention

0
Remix

Write multi-head self-attention from scratch: project the input to queries, keys and values, split them into heads, compute scaled dot-product attention, combine the heads and apply the output projection. Get the shapes and transposes right, and add a causal mask so a position cannot attend to later positions. Be ready to explain the scaling factor and the cost in sequence length.

AppleMetaUberTechnical phone screenLast asked Sep 2026
No answer yet. Write the first one; the best answer gets endorsed.