Implement multi-head self-attention
Write multi-head self-attention from scratch: project the input to queries, keys and values, split them into heads, compute scaled dot-product attention, combine the heads and apply the output projection. Get the shapes and transposes right, and add a causal mask so a position cannot attend to later positions. Be ready to explain the scaling factor and the cost in sequence length.
No answer yet. Write the first one; the best answer gets endorsed.


