Design large-scale model inference serving with back-of-envelope sizing

Design an inference system for a model used by 100 million daily active users. Do the back-of-envelope estimates for memory and bandwidth, then explain how you would reduce latency, avoid out-of-memory failures, batch requests and scale the fleet.