RMSNorm versus LayerNorm in a transformer block
Write the formulas for LayerNorm and RMSNorm and say what each normalizes. What does RMSNorm give up by skipping the mean, why does it work about as well in practice, and why do most recent LLMs use it? Where does the norm sit in a pre-norm versus post-norm block, and how does that choice affect training stability at depth?
Mistral AITechnical phone screenNo answer yet. Write the first one; the best answer gets endorsed.