Design ChatGPT

Design a consumer chat product backed by a large language model: conversations, streamed replies, history across devices, at 100M weekly users.

Functional requirements

• Users chat with a model in conversations; responses stream token by token.
• Conversations persist and can be resumed on any device.
• Users can stop generation, regenerate, and edit an earlier message (branching).

Non-functional requirements

• Time to first token under 1 s p50; streaming at 30+ tokens/s.
• 100M weekly users, so the GPU fleet is the cost that matters.
• Degrade gracefully at peak: queue or route to a smaller model rather than fail.

Back-of-the-envelope estimation

• 100M WAU, ~10 messages/user/week → ~1.6k messages/s average, ~5k/s peak.
• 500 output tokens per reply → 2.5M tokens/s at peak. At ~2.5k tokens/s per GPU (batched) that's ~1,000 GPUs just for decode, before redundancy.

• POST /conversations/{id}/messages {content, parent_id} → returns a stream (SSE) of {delta} events, then {done, message_id}.
• POST /messages/{id}/stop, GET /conversations/{id} (tree of messages).

• conversations(id, user_id, title, created_at)
• messages(id, conversation_id, parent_id, role, content, model, tokens, created_at): a tree, so edits create siblings.
• Store in a sharded SQL or wide-column store keyed by conversation_id; recent conversations cached.

flowchart LR
  U[Client] -->|SSE| API[Chat API]
  API --> CS[(Conversation store)]
  API --> ORCH[Orchestrator]
  ORCH --> MOD[Moderation]
  ORCH --> Q[Request router / queue]
  Q --> INF[Inference fleet]
  INF -->|tokens| ORCH

1. The chat API loads the conversation branch, builds the prompt (system prompt + truncated history), and checks moderation.
2. The router picks a model replica with free KV-cache capacity, preferring one that already holds this conversation's prefix.
3. Tokens stream back through the orchestrator to the client over SSE; the final message is written once at the end (plus periodic checkpoints).

Streaming. SSE over HTTP/2 is simpler than WebSockets and passes through proxies. The orchestrator holds the stream; if the client disconnects, generation keeps going and the client resumes by reading the saved message.

Context length. Long chats overflow the window. Keep the last N turns verbatim and summarize older turns into a running summary stored on the conversation.

GPU efficiency. Continuous batching, paged KV cache, and prefix caching for the shared system prompt (see the LLM serving question). Route by conversation for cache hits.

Overload. Admission control per user tier; free users queue or get a smaller model at peak; paid users get reserved capacity.

Safety. Input and output moderation run in parallel with generation; output is checked in chunks and the stream is cut if flagged.

• Writing messages only at the end saves writes but risks losing a long reply on crash; checkpoint every few seconds.
• Sticky routing improves cache hits but unbalances load; fall back to least-loaded when a replica is hot.