Data-parallel and fully sharded matrix multiply over simulated devices

0
Remix

Simulated devices can only exchange tensors through send and receive calls. Finish a data-parallel matrix multiply, then implement a fully sharded version where each device holds only a slice of the weights and gathers what it needs, and reason about the communication each one costs.

xAITechnical phone screenHardLast asked Apr 2026
No answer yet. Write the first one; the best answer gets endorsed.