Find the planted bugs in a small Transformer training script
A small Transformer, its training script and some tests are provided, with a few bugs hidden in them. Find and fix them so the loss goes down and the outputs are correct, and explain how each bug showed itself and how you tracked it down.
No answer yet. Write the first one; the best answer gets endorsed.