Design an LLM inference serving platform

Design the system that serves several large models to many customers with low latency and high GPU utilization.

Functional requirements

• Serve several models (small, medium, large) behind one API, with streaming.
• Support different customers with rate limits and priorities.
• Roll out new model versions without downtime.

Non-functional requirements

• Maximize tokens per GPU-second while keeping time to first token (TTFT) under 500 ms p50 and inter-token latency under 50 ms.
• High availability across zones; a GPU failure loses at most the in-flight requests on it.

Back-of-the-envelope estimation

• A 70B model in FP8 needs ~70 GB of weights: one 8×H100 node with tensor parallelism, leaving ~500 GB for KV cache.
• KV cache per token ≈ 2 × layers × kv_heads × head_dim × bytes ≈ 160 KB for a 70B GQA model → ~3M tokens of cache per node.

POST /v1/generate {model, messages, max_tokens, stream, priority} → SSE stream. Internal: replica.submit(request) on a scheduler queue.

Mostly in memory: per-replica queues, KV-cache block tables, prefix hash → block mapping. A model registry (versions, weights location, replica counts) in a small database.

flowchart LR
  C[Clients] --> GW[Gateway: auth, limits]
  GW --> RT[Router]
  RT --> P1[Replica pool: model A]
  RT --> P2[Replica pool: model B]
  P1 --> S1[Scheduler + engine]
  REG[(Model registry)] -. config .-> RT

Continuous batching. Instead of fixed batches, the scheduler adds new requests to the running batch every decode step and removes finished ones, so the GPU stays full.

Paged KV cache. Allocate KV memory in fixed blocks (like OS pages) so sequences of unknown length don't fragment memory; preempt or swap the lowest-priority sequence when blocks run out.

Prefill vs decode. Prefill is compute-bound, decode is memory-bandwidth-bound. Chunk long prefills so they don't stall decode, or disaggregate: separate prefill and decode pools that hand off the KV cache.

Prefix caching. Hash prompt prefixes (system prompts, few-shot examples) and reuse their KV blocks; the router sends requests to the replica that holds the prefix.

Speculative decoding. A small draft model proposes k tokens; the big model verifies them in one pass. 2–3× faster decode when acceptance is high.

Autoscaling. Scale on queue depth and KV-cache utilization, not CPU. Cold starts are minutes (loading weights), so keep warm headroom and pre-stage weights on local NVMe.

Rollouts. Shadow traffic then a canary percentage per model version; compare quality evals and latency before shifting traffic.

• Bigger batches raise throughput but hurt latency; set per-tier latency targets and cap batch size per pool.
• Disaggregated prefill/decode improves both but adds KV transfer cost over the network (needs fast interconnect).