Batched LLM inference API over GPUs
Design an inference API that batches requests across GPUs efficiently. The discussion goes deep on a few axes: utilization, scaling under spikes, and separating prompt processing from token generation.
Sign in to see the worked answer
The full answer, chapter by chapter, with what interviewers expect at each level. Free with an account: one full answer per company.