0

Batched LLM inference API over GPUs

by Frontier Interview· about 3 hours ago· ⑂ 0 remixes

Question: Batched LLM inference API over GPUs

Companies: Anthropic

Difficulty: Hard

Problem statement

Design an inference API that batches requests across GPUs efficiently. The discussion goes deep on a few axes: utilization, scaling under spikes, and separating prompt processing from token generation.

Requirements

An API that serves one large model (say 70B parameters) to many customers, batching their requests on a GPU fleet. The interview goes deep on three axes: keeping GPUs busy, surviving traffic spikes, and splitting prompt processing (prefill) from token generation (decode). Settle first: interactive streaming only, or also an asynchronous batch tier (both: the batch tier is what fills idle capacity)? Prompt and output lengths (assume 2,000 in, 500 out on average)? Latency targets per tier?

Functional requirements

  • POST /v1/generate: prompt in, streamed tokens out.
  • POST /v1/batches: a file of requests, results within 24 hours at a lower price.
  • Per-customer rate limits and priorities; cancellation when a client disconnects.
  • Usage metering (input and output tokens) per request.

Non-functional requirements

  • Interactive: time to first token (TTFT) p50 under 500 ms, p99 under 2 s; inter-token latency under 50 ms (20+ tokens/s per stream).
  • Throughput: maximize tokens per GPU-second. GPUs are most of the cost.
  • Spikes of 3× within minutes without failing paid traffic; degrade the batch tier first.
  • A GPU or node failure loses only the requests running on it, which are retried.

Back-of-the-envelope estimation

One serving node: 8 GPUs with 80 GB each, tensor-parallel. Per GPU: 989 TFLOP/s BF16, 3.35 TB/s HBM.

  • Weights: 70B in FP8 = 70 GB across the node. Free for KV cache after activations and overhead: ~500 GB.
  • KV cache per token for a 70B model with grouped-query attention (80 layers, 8 KV heads of width 128, BF16): 2 × 80 × 8 × 128 × 2 B = 320 KB. 500 GB / 320 KB ≈ 1.5M tokens of cache, so ~600 concurrent sequences at 2,500 tokens each.
  • Decode step at batch 600: read the weights once (70 GB / 26.8 TB/s = 2.6 ms) plus every sequence's KV cache (600 × 2,500 × 320 KB = 480 GB, 18 ms). About 20 ms per step: 30k tokens/s per node, 50 tokens/s per stream. Decode is memory-bound, and at long context the KV reads dominate, not the weights.
  • Prefill of a 2,000-token prompt: 2 × 70e9 × 2,000 = 2.8e14 FLOPs. The node delivers ~4e15 FLOP/s at 50% utilization: 70 ms per prompt, ~14 prompts/s per node. Prefill is compute-bound.

Fleet for 2,000 requests/s at peak:

  • Decode: 2,000 × 500 = 1M tokens/s ÷ 30k ≈ 34 nodes.
  • Prefill: 2,000 × 2,000 = 4M prompt tokens/s ÷ 28k tokens/s per node ≈ 143 nodes; with a 50% prefix-cache hit rate (shared system prompts), ~72 nodes.
  • Prefill and decode need very different amounts of hardware for the same traffic. That ratio is the argument for separate pools later.

API design

POST /v1/generate
  {model, messages, max_tokens, temperature, stream: true, priority: "interactive"}
  → SSE: {type: "token", text} ... {type: "done", usage: {input_tokens, output_tokens}, stop_reason}

POST /v1/batches  {input_file_id, model}         → {batch_id, status: "queued"}
GET  /v1/batches/{id}                             → {status, counts, output_file_id}

Internal, gateway to scheduler: Submit(request_id, tenant, priority, prompt_tokens, max_tokens, deadline) and a stream back of token chunks. Cancel(request_id) when the client disconnects, which frees the sequence's KV blocks at once.

Data model

Almost all state is in memory on the serving nodes:

  • Per replica: the running batch, a waiting queue per priority, the KV block table (sequence → list of 16-token blocks), and a prefix index (hash of each full block of prompt tokens → block id) for reuse.
  • Router: per replica load (queue depth, free KV blocks, running sequences) refreshed every 100 ms, and which prefixes each replica holds (a compact sketch).
  • Durable: tenant limits and priorities (config store), usage records (append-only log to billing), batch jobs (batch_id, tenant, status, input_ref, output_ref, counts) and their items in object storage.

High-level design

Build it one requirement at a time. Each drawing adds to the one before; the shaded boxes are what that step adds.

1) One replica with continuous batching

The gateway authenticates, applies rate limits and meters usage. The replica's scheduler runs continuous batching: every decode step it removes finished sequences and admits waiting ones, so the batch stays full even though requests have different lengths. With static batching, a batch waits for its longest member, and GPUs idle on the padding.

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Clients"}},
 {"type":"rectangle","id":"gw","x":260,"y":0,"width":180,"height":70,"label":{"text":"API gateway"}},
 {"type":"rectangle","id":"sched","x":520,"y":0,"width":180,"height":70,"label":{"text":"Scheduler"}},
 {"type":"rectangle","id":"engine","x":780,"y":0,"width":180,"height":70,"label":{"text":"GPU engine"}},
 {"type":"ellipse","id":"kv","x":770,"y":160,"width":200,"height":80,"label":{"text":"KV cache"}},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":80,"height":0,"start":{"id":"client"},"end":{"id":"gw"}},
 {"type":"arrow","id":"a2","x":440,"y":35,"width":80,"height":0,"start":{"id":"gw"},"end":{"id":"sched"}},
 {"type":"arrow","id":"a3","x":700,"y":35,"width":80,"height":0,"start":{"id":"sched"},"end":{"id":"engine"}},
 {"type":"arrow","id":"a4","x":870,"y":70,"width":1,"height":90,"start":{"id":"engine"},"end":{"id":"kv"}}
]}

2) Many GPUs behind a router

Replicas are independent model copies. The router picks one per request by load (free KV blocks and queue depth, not request count, since requests differ 100× in cost) and by prefix affinity: a request whose system prompt is cached on replica 7 goes there if replica 7 is not overloaded, saving its prefill.

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-60,"text":"Shaded: added in this step","fontSize":16},
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Clients"}},
 {"type":"rectangle","id":"gw","x":260,"y":0,"width":180,"height":70,"label":{"text":"API gateway"}},
 {"type":"rectangle","id":"router","x":520,"y":0,"width":180,"height":70,"label":{"text":"Router"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"r1","x":780,"y":0,"width":180,"height":70,"label":{"text":"Replica 1"}},
 {"type":"rectangle","id":"r2","x":780,"y":140,"width":180,"height":70,"label":{"text":"Replica N"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":80,"height":0,"start":{"id":"client"},"end":{"id":"gw"}},
 {"type":"arrow","id":"a2","x":440,"y":35,"width":80,"height":0,"start":{"id":"gw"},"end":{"id":"router"}},
 {"type":"arrow","id":"a3","x":700,"y":35,"width":80,"height":0,"start":{"id":"router"},"end":{"id":"r1"}},
 {"type":"arrow","id":"a4","x":700,"y":55,"width":80,"height":120,"start":{"id":"router"},"end":{"id":"r2"}}
]}

3) Separate prefill from decode

A long prefill inside a decode batch stalls every stream in that batch for its duration (70 ms for 2,000 tokens, more than one inter-token budget). Two fixes, in order of complexity:

  • Chunked prefill: split each prompt into 512-token chunks and mix one chunk into each decode step. Decode streams see a bounded hiccup, and one pool handles both.
  • Disaggregation: a prefill pool computes the prompt's KV cache and ships it to a decode pool over the node interconnect. Each pool is sized for its own bottleneck (compute for prefill, memory for decode), and each can use a different parallelism layout.
{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-60,"text":"Shaded: added in this step","fontSize":16},
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Clients"}},
 {"type":"rectangle","id":"gw","x":260,"y":0,"width":180,"height":70,"label":{"text":"API gateway"}},
 {"type":"rectangle","id":"router","x":520,"y":0,"width":180,"height":70,"label":{"text":"Router"}},
 {"type":"rectangle","id":"pre","x":780,"y":0,"width":180,"height":70,"label":{"text":"Prefill pool"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"dec","x":1120,"y":0,"width":180,"height":70,"label":{"text":"Decode pool"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":80,"height":0,"start":{"id":"client"},"end":{"id":"gw"}},
 {"type":"arrow","id":"a2","x":440,"y":35,"width":80,"height":0,"start":{"id":"gw"},"end":{"id":"router"}},
 {"type":"arrow","id":"a3","x":700,"y":35,"width":80,"height":0,"start":{"id":"router"},"end":{"id":"pre"}},
 {"type":"arrow","id":"a4","x":960,"y":35,"width":160,"height":0,"start":{"id":"pre"},"end":{"id":"dec"},"label":{"text":"KV cache"}},
 {"type":"arrow","id":"a5","x":1210,"y":70,"width":-1120,"height":0,"points":[[0,0],[0,90],[-1120,90],[-1120,0]],"start":{"id":"dec"},"end":{"id":"client"},"strokeStyle":"dashed","label":{"text":"token stream"}}
]}

4) Spikes, priorities and the batch tier

Admission control in front of the router holds a queue per priority. Interactive requests go first; batch-tier work is admitted only when free KV capacity is above a watermark, so it soaks up idle GPU time and is the first thing shed in a spike. An autoscaler watches queue time and KV utilization (not CPU) and adds replicas from a warm pool.

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-60,"text":"Shaded: added in this step","fontSize":16},
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Clients"}},
 {"type":"rectangle","id":"gw","x":260,"y":0,"width":180,"height":70,"label":{"text":"API gateway"}},
 {"type":"rectangle","id":"router","x":520,"y":0,"width":180,"height":70,"label":{"text":"Admission +\nrouter"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"pre","x":780,"y":0,"width":180,"height":70,"label":{"text":"Prefill pool"}},
 {"type":"rectangle","id":"dec","x":1120,"y":0,"width":180,"height":70,"label":{"text":"Decode pool"}},
 {"type":"rectangle","id":"batch","x":260,"y":-200,"width":180,"height":70,"label":{"text":"Batch jobs"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"auto","x":950,"y":-200,"width":180,"height":70,"label":{"text":"Autoscaler"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":80,"height":0,"start":{"id":"client"},"end":{"id":"gw"}},
 {"type":"arrow","id":"a2","x":440,"y":35,"width":80,"height":0,"start":{"id":"gw"},"end":{"id":"router"}},
 {"type":"arrow","id":"a3","x":700,"y":35,"width":80,"height":0,"start":{"id":"router"},"end":{"id":"pre"}},
 {"type":"arrow","id":"a4","x":960,"y":35,"width":160,"height":0,"start":{"id":"pre"},"end":{"id":"dec"},"label":{"text":"KV cache"}},
 {"type":"arrow","id":"a5","x":1210,"y":70,"width":-1120,"height":0,"points":[[0,0],[0,90],[-1120,90],[-1120,0]],"start":{"id":"dec"},"end":{"id":"client"},"strokeStyle":"dashed","label":{"text":"token stream"}},
 {"type":"arrow","id":"b1","x":440,"y":-165,"width":170,"height":165,"start":{"id":"batch"},"end":{"id":"router"},"label":{"text":"low priority"}},
 {"type":"arrow","id":"s1","x":990,"y":-130,"width":-120,"height":130,"start":{"id":"auto"},"end":{"id":"pre"},"strokeStyle":"dashed"},
 {"type":"arrow","id":"s2","x":1090,"y":-130,"width":120,"height":130,"start":{"id":"auto"},"end":{"id":"dec"},"strokeStyle":"dashed"}
]}

Final design

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Clients"}},
 {"type":"rectangle","id":"gw","x":260,"y":0,"width":180,"height":70,"label":{"text":"API gateway"}},
 {"type":"rectangle","id":"router","x":520,"y":0,"width":180,"height":70,"label":{"text":"Admission +\nrouter"}},
 {"type":"rectangle","id":"pre","x":780,"y":0,"width":180,"height":70,"label":{"text":"Prefill pool"}},
 {"type":"rectangle","id":"dec","x":1120,"y":0,"width":180,"height":70,"label":{"text":"Decode pool"}},
 {"type":"rectangle","id":"batch","x":260,"y":-200,"width":180,"height":70,"label":{"text":"Batch jobs"}},
 {"type":"rectangle","id":"auto","x":950,"y":-200,"width":180,"height":70,"label":{"text":"Autoscaler"}},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":80,"height":0,"start":{"id":"client"},"end":{"id":"gw"}},
 {"type":"arrow","id":"a2","x":440,"y":35,"width":80,"height":0,"start":{"id":"gw"},"end":{"id":"router"}},
 {"type":"arrow","id":"a3","x":700,"y":35,"width":80,"height":0,"start":{"id":"router"},"end":{"id":"pre"}},
 {"type":"arrow","id":"a4","x":960,"y":35,"width":160,"height":0,"start":{"id":"pre"},"end":{"id":"dec"},"label":{"text":"KV cache"}},
 {"type":"arrow","id":"a5","x":1210,"y":70,"width":-1120,"height":0,"points":[[0,0],[0,90],[-1120,90],[-1120,0]],"start":{"id":"dec"},"end":{"id":"client"},"strokeStyle":"dashed","label":{"text":"token stream"}},
 {"type":"arrow","id":"b1","x":440,"y":-165,"width":170,"height":165,"start":{"id":"batch"},"end":{"id":"router"},"label":{"text":"low priority"}},
 {"type":"arrow","id":"s1","x":990,"y":-130,"width":-120,"height":130,"start":{"id":"auto"},"end":{"id":"pre"},"strokeStyle":"dashed"},
 {"type":"arrow","id":"s2","x":1090,"y":-130,"width":120,"height":130,"start":{"id":"auto"},"end":{"id":"dec"},"strokeStyle":"dashed"},
 {"type":"text","id":"note","x":0,"y":230,"text":"Solid: request path. Dashed: token stream and scaling control.","fontSize":16}
]}

Deep dives

Continuous batching and the KV cache (utilization). The KV cache, not compute, limits how many sequences fit. Allocate it in fixed blocks of 16 tokens through a block table per sequence (paged attention), so memory is not reserved for max_tokens up front: a sequence that stops after 50 tokens never held space for 4,000. Fragmentation drops from ~60% wasted with contiguous reservation to under 5%. When blocks run out mid-generation, preempt the lowest-priority sequence: drop its cache and recompute later (cheap for short prompts) or swap it to CPU memory (better for long ones).

Batch size versus latency. Decode step time grows with batch size: ~2.6 ms of weight reads plus KV reads proportional to batch × context. At 600 long sequences a step is ~20 ms (50 tokens/s per stream, 30k/s per node); at 100 sequences it is ~6 ms (about 170 tokens/s each, ~17k/s per node). Run interactive pools at the batch size that meets the inter-token target with margin, and run the batch tier at the largest batch memory allows. Same hardware, two operating points.

Prefill/decode split, sized with numbers. From the estimate: ~72 prefill nodes and ~34 decode nodes at peak. Moving a 2,000-token KV cache (2,000 × 320 KB = 640 MB) between nodes at 400 Gb/s (~50 GB/s) takes about 13 ms, small next to a 70 ms prefill and well within the TTFT budget. Transfer layer by layer while prefill runs so the decode node can start almost at once. If the interconnect is slow, chunked prefill in a unified pool is the safer choice.

Prefix caching. Many requests share a long system prompt or few-shot block. Hash each full 16-token block of the prompt (chained with the previous block's hash, so a hash identifies the whole prefix) and reuse matching blocks with reference counts; evict least-recently-used blocks with zero references. At a 50% hit rate, prefill compute halves and TTFT drops in proportion. Routing by prefix affinity is what makes the hit rate high; without it, the same prompt is cached on every replica.

Spikes (scaling). New capacity is slow: loading 70 GB of weights takes 1–3 minutes even from local NVMe, longer from object storage. So:

  1. Keep warm headroom (~20–30%) sized to the spikes seen in the last weeks.
  2. Shed in order: pause batch-tier admission, then cap max_tokens for free-tier requests, then return 429 with Retry-After per tenant above its share. Never accept a request that cannot meet its deadline: queueing it just turns a fast failure into a slow one.
  3. Scale on leading signals: queue wait time and KV utilization above 85%, not GPU utilization (always near 100% under continuous batching).
  4. Spread tenants' limits so one customer's spike cannot take the whole fleet (per-tenant token-bucket limits on tokens, not requests).

Failure handling. A node failure kills its running sequences. The gateway still holds the request and the tokens already streamed, so it resubmits with the output so far appended to the prompt; the client sees a pause, not an error. Health checks run a tiny generation every few seconds, because a GPU can fail in ways that leave the process up and the output wrong.

Batch tier. Batch jobs are split into items and fed through the same admission path at low priority. Items are idempotent by (batch_id, item_index), so a preempted item is simply requeued. The 24-hour window lets the scheduler fill the overnight trough: if daytime peak utilization is 70% of fleet capacity and the trough is 30%, batch work can use the other 40–70% of the day's capacity at no extra hardware cost, which is why it can be sold cheaper.

Trade-offs

  • Large batches maximize tokens per GPU but slow every stream; serve two operating points instead of one compromise.
  • Disaggregation sizes each pool for its bottleneck and stops prefill from stalling decode, but adds KV transfer, two schedulers and more failure modes; chunked prefill gets much of the benefit with one pool.
  • Prefix-affinity routing raises cache hits but can concentrate load; cap the affinity when a replica's queue is long.
  • Preemption by recompute is simple and frees memory at once but wastes the prefill work; swapping keeps it at the cost of PCIe bandwidth.
  • Warm headroom costs money every hour to avoid failing during spikes; the batch tier recovers most of it by using that headroom when it is idle.

Interviewer expectations

Mid-level (L4)

  • Puts a gateway, a router and GPU replicas behind the API, with streaming and rate limits.
  • Explains batching for throughput and the latency it costs.
  • Mentions autoscaling and the slow start of model replicas.

Senior (L5)

  • Explains continuous batching and the paged KV cache, and computes KV memory per token and concurrent sequences.
  • Separates compute-bound prefill from memory-bound decode, with chunked prefill or disaggregation, and routes by load and prefix.
  • Designs spike handling: admission control, priority tiers, shedding order, scaling on queue time and KV utilization.

Staff (L6+)

  • Sizes prefill and decode pools from traffic numbers and checks KV transfer cost against the TTFT budget.
  • Uses the batch tier to fill troughs and explains the pricing logic, and plans failure recovery that keeps streams alive.
  • Names the operating points (batch size versus inter-token latency) and the metrics to run the fleet by: tokens per GPU-second, TTFT and inter-token percentiles, KV utilization, shed rate.

Sign in to comment, vote or remix.