Scaled dot-product attention from scratch
Write scaled dot-product attention by hand: from queries, keys and values, compute the scores, scale them by the square root of the key size, apply an optional mask, take a softmax and weight the values. Start from the formula, keep the tensor shapes straight, and extend to several heads if time allows. The code doesn't need to run.
No answer yet. Write the first one; the best answer gets endorsed.
