Adam, AdamW and decoupled weight decay

0
Remix

Write Adam's update: the first and second moment estimates, bias correction and the step. Why is adding an L2 penalty to the loss not the same as weight decay once Adam's per-parameter scaling is involved, and what does AdamW change? Which parameters usually get no weight decay in a transformer, and how does the learning-rate schedule interact with the decay?

Mistral AITechnical phone screenLast asked Sep 2026
No answer yet. Write the first one; the best answer gets endorsed.