Mixture of experts and expert parallelism
Explain how a mixture-of-experts transformer layer works and follow a batch of tokens through it: the router, top-k selection, the all-to-all exchange that sends tokens to experts on other devices, the computation and the return trip. Why does expert parallelism exist, what limits it (capacity factors, dropped tokens, load imbalance, communication), and how do auxiliary losses keep the experts evenly used?
Mistral AITechnical phone screenNo answer yet. Write the first one; the best answer gets endorsed.