Attention and a small Transformer with a hand-written backward pass

Implement scaled dot-product attention and a small autoregressive Transformer block from scratch, including the causal mask. Then write the backward pass by hand (no autograd) for the pieces the interviewer picks, and check your gradients numerically.