Dense versus mixture-of-experts at fixed compute

With a fixed activated-parameter budget, compare a dense model against a mixture-of-experts on training cost, memory, communication, stability, inference and engineering effort, and recommend based on team context. Reported on 1point3acres' Hugging Face bank; reconstructed in our own words.