Design ChatGPT streaming with resumable responses and long conversations

This question is usually the second half of "Design ChatGPT": the interviewer skips the basics and pushes on three follow-ups. Confirm that framing in the first minute, then spend the time on the three hard parts.

Functional requirements

• A reply streams to the client token by token over Server-Sent Events (SSE).
• If the connection drops (a train tunnel, a laptop lid, a phone switching from Wi-Fi to LTE), the client reconnects and continues from the last token it showed. No token is lost, none is shown twice, and generation is not restarted.
• A second device that opens the conversation mid-reply sees the same reply streaming.
• The user can stop a reply; the partial text is kept.
• Conversations can be very long (hundreds of turns, more tokens than the model's context window) and still answer well.

Non-functional requirements

• Time to first token under 1 s at p50 for short prompts; the stream itself adds under 100 ms over the model.
• A finished reply is durable: it is never lost once the user has seen it complete.
• Generation cost is the dominant cost; the streaming layer must not cause re-generation.
• Scale: 100M weekly users, about 5k new replies per second at peak.

Back-of-the-envelope estimation

• Concurrent streams. 5k replies/s × 20 s average streaming time (500 tokens at 25 tokens/s) = 100k open streams at peak.
• Token rate. 5k × 500 = 2.5M tokens/s. Sending one SSE event per token would be 2.5M events/s; batching to one event every 50 ms (about 1–2 tokens each at 25 tokens/s) gives 100k × 20 = 2M events/s. Batching to 100 ms halves it. A human cannot see the difference between 50 and 100 ms.
• Gateway connections. One gateway pod holds ~20k idle-ish SSE connections comfortably (memory per connection ~20–50 KB with HTTP/2 buffers). 100k streams plus 2× headroom and a pod-loss margin: about 15 pods per region.
• Stream log memory. One reply ≈ 500 tokens × 4 bytes ≈ 2 KB of text, ~10 KB with per-entry overhead at 100 ms batches. Keep finished replies 10 minutes for late resumes: 5k/s × 600 s = 3M replies × 10 KB = 30 GB, plus 1 GB for the active ones. A handful of Redis shards.
• Long context cost. A 100k-token conversation re-sent every turn: at ~10k tokens/s of prefill per GPU, that is 10 GPU-seconds per turn before the first output token, and a TTFT of several seconds even on many GPUs. Prefix caching and summaries (below) are not optional at this length.

• POST /v1/conversations/{cid}/messages {content, parent_id, client_msg_id} → 202 {message_id, response_id}. client_msg_id makes a retried POST idempotent: the same id returns the same response_id instead of starting a second generation.
• GET /v1/responses/{response_id}/stream with header Last-Event-ID: <seq> (optional) → text/event-stream. Events:
  • id: 42 event: delta data: {"text": "..."}
  • event: done data: {"message_id", "finish_reason", "usage"}
  • event: error data: {"code", "retryable"}
  • a comment line : ping every 15 s so proxies do not close an idle connection.
• POST /v1/responses/{response_id}/stop → 202.
• GET /v1/conversations/{cid} → the message tree, with any in-progress reply marked status: streaming and its response_id.

Splitting "start" from "stream" is the key API decision. The POST is a short, retryable request; the stream is a read that can be repeated any number of times. That is what makes resume possible.

• conversations(id, user_id, title, summary, summary_through_msg_id, updated_at), sharded by user_id.
• messages(id, conversation_id, parent_id, role, content, status, model, tokens, created_at): a tree, so an edit or a regenerate adds a sibling. status is streaming | complete | stopped | failed.
• Stream log (Redis Stream per reply): key resp:{response_id}, entries {seq, text} in order, plus a state hash {status, owner_worker, last_seq, cancel}. TTL 10 minutes after done.

Built in three steps. Each drawing adds to the one before; shaded parts are new in that step.

1) Stream a reply

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Client"}},
 {"type":"rectangle","id":"gw","x":340,"y":0,"width":180,"height":70,"label":{"text":"Chat API +\nSSE gateway"}},
 {"type":"rectangle","id":"gen","x":680,"y":0,"width":180,"height":70,"label":{"text":"Generation\nworker"}},
 {"type":"rectangle","id":"inf","x":1020,"y":0,"width":180,"height":70,"label":{"text":"Inference\nfleet"}},
 {"type":"ellipse","id":"conv","x":660,"y":200,"width":220,"height":80,"label":{"text":"Conversation\nstore"}},
 {"type":"arrow","id":"c-gw","x":180,"y":25,"width":160,"height":0,"start":{"id":"client"},"end":{"id":"gw"},"label":{"text":"POST"}},
 {"type":"arrow","id":"gw-gen","x":520,"y":25,"width":160,"height":0,"start":{"id":"gw"},"end":{"id":"gen"},"label":{"text":"job"}},
 {"type":"arrow","id":"gen-inf","x":860,"y":25,"width":160,"height":0,"start":{"id":"gen"},"end":{"id":"inf"},"label":{"text":"prompt"}},
 {"type":"arrow","id":"inf-gen","x":1020,"y":55,"width":-160,"height":0,"start":{"id":"inf"},"end":{"id":"gen"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gen-gw","x":680,"y":55,"width":-160,"height":0,"start":{"id":"gen"},"end":{"id":"gw"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gw-c","x":340,"y":55,"width":-160,"height":0,"start":{"id":"gw"},"end":{"id":"client"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gen-conv","x":770,"y":70,"width":1,"height":130,"start":{"id":"gen"},"end":{"id":"conv"},"label":{"text":"history"}},
 {"type":"text","id":"legend","x":0,"y":330,"text":"Black: request path. Red dashed: token stream (SSE).","fontSize":16}
]}

1. The client POSTs the message. The chat API writes the user message, creates the assistant message with status: streaming, and hands a job to a generation worker.
2. The worker loads the conversation branch, builds the prompt, and calls the inference fleet, which streams tokens back.
3. Tokens flow back through the gateway to the client over SSE.

This works until the client's connection drops: the tokens produced while it was gone have nowhere to go, and the only way to recover is to regenerate. That is the problem the next step solves.

2) Survive disconnects: a stream log between generation and delivery

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Client"}},
 {"type":"rectangle","id":"gw","x":340,"y":0,"width":180,"height":70,"label":{"text":"Chat API +\nSSE gateway"}},
 {"type":"rectangle","id":"gen","x":680,"y":0,"width":180,"height":70,"label":{"text":"Generation\nworker"}},
 {"type":"rectangle","id":"inf","x":1020,"y":0,"width":180,"height":70,"label":{"text":"Inference\nfleet"}},
 {"type":"ellipse","id":"conv","x":660,"y":200,"width":220,"height":80,"label":{"text":"Conversation\nstore"}},
 {"type":"ellipse","id":"log","x":320,"y":200,"width":220,"height":80,"label":{"text":"Stream log\n(Redis)"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"c-gw","x":180,"y":25,"width":160,"height":0,"start":{"id":"client"},"end":{"id":"gw"},"label":{"text":"POST"}},
 {"type":"arrow","id":"gw-gen","x":520,"y":25,"width":160,"height":0,"start":{"id":"gw"},"end":{"id":"gen"},"label":{"text":"job"}},
 {"type":"arrow","id":"gen-inf","x":860,"y":25,"width":160,"height":0,"start":{"id":"gen"},"end":{"id":"inf"},"label":{"text":"prompt"}},
 {"type":"arrow","id":"inf-gen","x":1020,"y":55,"width":-160,"height":0,"start":{"id":"inf"},"end":{"id":"gen"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gw-c","x":340,"y":55,"width":-160,"height":0,"start":{"id":"gw"},"end":{"id":"client"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gen-conv","x":770,"y":70,"width":1,"height":130,"start":{"id":"gen"},"end":{"id":"conv"},"label":{"text":"history"}},
 {"type":"arrow","id":"gen-log","x":700,"y":70,"width":-192,"height":142,"start":{"id":"gen"},"end":{"id":"log"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"log-gw","x":430,"y":200,"width":1,"height":-130,"start":{"id":"log"},"end":{"id":"gw"},"strokeColor":"#e03131","strokeStyle":"dashed","label":{"text":"tail from seq"}},
 {"type":"text","id":"legend","x":0,"y":330,"text":"Shaded: added in this step. Black: request path. Red dashed: token stream.","fontSize":16}
]}

1. The worker no longer streams to a connection. It appends batches of tokens to the reply's stream log with increasing sequence numbers (XADD resp:{id} every 50–100 ms) and keeps generating whether or not anyone is listening.
2. The gateway serves GET /stream by reading the log from Last-Event-ID + 1 (replay), then blocking on new entries (tail). Each SSE event carries id: <seq>, so the browser's EventSource sends the right Last-Event-ID on reconnect by itself.
3. Any gateway pod can serve any stream, so a reconnect can land anywhere: no sticky sessions. A second device is just a second reader of the same log.
4. On done the worker writes the full message to the conversation store (status: complete) and then marks the log done. The log expires 10 minutes later; a client that comes back after that reads the stored message instead.

3) Long conversations: summaries and prefix reuse

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Client"}},
 {"type":"rectangle","id":"gw","x":340,"y":0,"width":180,"height":70,"label":{"text":"Chat API +\nSSE gateway"}},
 {"type":"rectangle","id":"gen","x":680,"y":0,"width":180,"height":70,"label":{"text":"Generation\nworker"}},
 {"type":"rectangle","id":"inf","x":1020,"y":0,"width":180,"height":70,"label":{"text":"Inference\nfleet"}},
 {"type":"ellipse","id":"conv","x":660,"y":200,"width":220,"height":80,"label":{"text":"Conversation\nstore"}},
 {"type":"ellipse","id":"log","x":320,"y":200,"width":220,"height":80,"label":{"text":"Stream log\n(Redis)"}},
 {"type":"rectangle","id":"summ","x":1020,"y":205,"width":180,"height":70,"label":{"text":"Summarizer"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"c-gw","x":180,"y":25,"width":160,"height":0,"start":{"id":"client"},"end":{"id":"gw"},"label":{"text":"POST"}},
 {"type":"arrow","id":"gw-gen","x":520,"y":25,"width":160,"height":0,"start":{"id":"gw"},"end":{"id":"gen"},"label":{"text":"job"}},
 {"type":"arrow","id":"gen-inf","x":860,"y":25,"width":160,"height":0,"start":{"id":"gen"},"end":{"id":"inf"},"label":{"text":"prompt"}},
 {"type":"arrow","id":"inf-gen","x":1020,"y":55,"width":-160,"height":0,"start":{"id":"inf"},"end":{"id":"gen"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gw-c","x":340,"y":55,"width":-160,"height":0,"start":{"id":"gw"},"end":{"id":"client"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gen-conv","x":770,"y":70,"width":1,"height":130,"start":{"id":"gen"},"end":{"id":"conv"},"label":{"text":"history"}},
 {"type":"arrow","id":"gen-log","x":700,"y":70,"width":-192,"height":142,"start":{"id":"gen"},"end":{"id":"log"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"log-gw","x":430,"y":200,"width":1,"height":-130,"start":{"id":"log"},"end":{"id":"gw"},"strokeColor":"#e03131","strokeStyle":"dashed","label":{"text":"tail from seq"}},
 {"type":"arrow","id":"summ-conv","x":1020,"y":240,"width":-140,"height":0,"start":{"id":"summ"},"end":{"id":"conv"},"strokeStyle":"dashed","label":{"text":"summary"}},
 {"type":"text","id":"legend","x":0,"y":330,"text":"Shaded: added in this step. Black dashed: async job. Red dashed: token stream.","fontSize":16}
]}

1. The worker builds the prompt to a token budget: system prompt, then the conversation's running summary of old turns, then as many recent turns as fit, verbatim, newest last.
2. A summarizer job runs off the request path when the verbatim history passes a threshold (say 60% of the window). It folds the oldest turns into the summary and records summary_through_msg_id.
3. The router sends the turn to a replica that likely still holds this conversation's KV cache (route by conversation id), so only the new turn is prefilled.

Final design

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":70,"label":{"text":"Client"}},
 {"type":"rectangle","id":"gw","x":340,"y":0,"width":180,"height":70,"label":{"text":"Chat API +\nSSE gateway"}},
 {"type":"rectangle","id":"gen","x":680,"y":0,"width":180,"height":70,"label":{"text":"Generation\nworker"}},
 {"type":"rectangle","id":"inf","x":1020,"y":0,"width":180,"height":70,"label":{"text":"Inference\nfleet"}},
 {"type":"ellipse","id":"conv","x":660,"y":200,"width":220,"height":80,"label":{"text":"Conversation\nstore"}},
 {"type":"ellipse","id":"log","x":320,"y":200,"width":220,"height":80,"label":{"text":"Stream log\n(Redis)"}},
 {"type":"rectangle","id":"summ","x":1020,"y":205,"width":180,"height":70,"label":{"text":"Summarizer"}},
 {"type":"arrow","id":"c-gw","x":180,"y":25,"width":160,"height":0,"start":{"id":"client"},"end":{"id":"gw"},"label":{"text":"POST"}},
 {"type":"arrow","id":"gw-gen","x":520,"y":25,"width":160,"height":0,"start":{"id":"gw"},"end":{"id":"gen"},"label":{"text":"job"}},
 {"type":"arrow","id":"gen-inf","x":860,"y":25,"width":160,"height":0,"start":{"id":"gen"},"end":{"id":"inf"},"label":{"text":"prompt"}},
 {"type":"arrow","id":"inf-gen","x":1020,"y":55,"width":-160,"height":0,"start":{"id":"inf"},"end":{"id":"gen"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gw-c","x":340,"y":55,"width":-160,"height":0,"start":{"id":"gw"},"end":{"id":"client"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"gen-conv","x":770,"y":70,"width":1,"height":130,"start":{"id":"gen"},"end":{"id":"conv"},"label":{"text":"history"}},
 {"type":"arrow","id":"gen-log","x":700,"y":70,"width":-192,"height":142,"start":{"id":"gen"},"end":{"id":"log"},"strokeColor":"#e03131","strokeStyle":"dashed"},
 {"type":"arrow","id":"log-gw","x":430,"y":200,"width":1,"height":-130,"start":{"id":"log"},"end":{"id":"gw"},"strokeColor":"#e03131","strokeStyle":"dashed","label":{"text":"tail from seq"}},
 {"type":"arrow","id":"summ-conv","x":1020,"y":240,"width":-140,"height":0,"start":{"id":"summ"},"end":{"id":"conv"},"strokeStyle":"dashed","label":{"text":"summary"}},
 {"type":"text","id":"legend","x":0,"y":330,"text":"Black: request path. Black dashed: async job. Red dashed: token stream.","fontSize":16}
]}

SSE, and why not WebSockets. The traffic is one request up, a long stream down: SSE fits exactly. It is plain HTTP, so it passes corporate proxies and CDNs, multiplexes over HTTP/2 (many tabs share one TCP connection, avoiding the HTTP/1.1 six-connections-per-host limit), and EventSource reconnects with Last-Event-ID for free. Gotchas worth naming: turn off proxy buffering (X-Accel-Buffering: no on nginx-style proxies, or the stream arrives in one lump at the end); send a heartbeat comment every 15 s because load balancers close connections idle for 60 s; set the retry: field (e.g. 1000 ms) and add jitter on the client so 100k clients do not reconnect in the same second after a gateway deploy. Native EventSource cannot send an Authorization header or a POST body, which is one reason to split POST (start) from GET (stream), and to authenticate the GET with a short-lived, response-scoped token or a cookie. WebSockets earn their place only when the client must send mid-stream (voice, collaborative editing).

Resume semantics: exactly-once on screen. Delivery is at least once (a reconnect can replay an event the client already rendered), so the client makes it exactly once: it remembers the highest seq it rendered and drops any event with seq <= last. Sequence numbers are assigned by the single writer (the worker that owns the reply), so there are no gaps or duplicates in the log. Batching: the worker appends every 50–100 ms or every 16 tokens, whichever comes first, so one reply is ~200 log entries, not 500. If the worker crashes, the reply's owner_worker lease (heartbeated every 2 s) expires; a new worker cannot continue the same sampling mid-sentence deterministically, so it marks the reply failed with retryable: true and the client offers "Regenerate". Saying this honestly beats claiming seamless failover.

Stop and abandoned streams. POST /stop sets cancel = 1 in the reply's state hash; the worker checks it between batches (≤100 ms), aborts the inference request (which frees the GPU slot and KV cache), writes the partial text with status: stopped, and emits done with finish_reason: stopped. The opposite case: nobody is listening. Keep generating if a reader disconnected less than ~30 s ago (likely a reconnect), but cancel when no reader has attached for 2 minutes and the client never acknowledged; otherwise a closed tab burns a full 2,000-token reply. At ~2% abandoned replies of 5k/s, that is 100 wasted generations per second.

Durability: when is a reply safe? The stream log is in memory with replication, not the system of record. The worker writes the final message to the conversation store before emitting done, so a client that saw done can always reload the message. For long replies (a 4,000-token answer takes over 2 minutes) also checkpoint the partial content to the store every ~10 s; if the Redis shard is lost, the reply degrades to "here is what we had" instead of vanishing.

Long conversations: the token budget. With a 128k-token window, a 2k system prompt, and 4k reserved for the reply, the history gets ~122k. Policy: newest turns verbatim until the budget is 70% used, then the running summary, then retrieval: embed old turns and pull the top 5 relevant ones for the new question (useful when a user asks "what did we decide about X" 300 turns later). The summary is rebuilt incrementally (old summary + the turns being folded in → new summary), never from scratch, so its cost is O(new turns). Show the user when context was condensed; silent forgetting reads as the model being dumb.

Long conversations: prefill cost and the KV cache. Turn 50 of a conversation re-sends turns 1–49. Without reuse, a 60k-token prompt costs ~6 GPU-seconds of prefill per turn (at ~10k tokens/s per GPU) and a multi-second TTFT. With prefix caching, the replica that served turn 49 still holds those 60k tokens' KV blocks (if it has room: a 70B-class model with grouped-query attention stores 80 layers × 8 KV heads × 128 dims × 2 (K and V) × 2 bytes ≈ 320 KB per token, so 60k tokens ≈ 20 GB, a quarter of an 80 GB GPU), so turn 50 only prefills the new message. That is why routing is by conversation id with a fallback to least-loaded, and why summaries matter: rewriting old turns changes the prefix and invalidates the cache, so fold turns in large, rare steps (every ~20k tokens), not every turn.

Multi-region. Keep a conversation's reply in one region: the worker, the log and the gateway readers are all in the user's home region. A reconnect that lands in another region (DNS failover, travel) is proxied to the home region's gateway by response_id, which encodes the region. Cross-region replication of the conversation store is asynchronous; a region loss can lose the last few seconds of in-flight replies, which the client sees as a retryable failure.

• "The user has the app open on a phone and a laptop. What does the laptop show mid-reply?" Conversation fetch shows status: streaming and the response_id; the laptop opens the same stream with no Last-Event-ID and replays from seq 1.
• "Why not let the gateway call the model directly and skip the log?" Then the reply lives and dies with one TCP connection; every mobile network change costs a regeneration, which at our token volume is real GPU money.
• "How do you test resume?" A chaos test that kills client connections at random points and checks the rendered text equals the stored message byte for byte.
• "Tool calls and images mid-stream?" Typed events (event: tool_call, event: image) in the same ordered log; the client renders by type, and resume works unchanged.
• "Rate-limit streaming?" Concurrency limit per user (e.g. 3 open generations), enforced at POST; the stream GET is free because it only reads.

• A stream log between generation and delivery buys resume, multi-device and gateway restarts without regeneration; it costs one more hop (a few ms) and ~30 GB of Redis per region.
• Batching tokens every 50–100 ms cuts events 10× versus per-token; the text appears in small chunks rather than letter by letter, which users do not notice at those intervals.
• Failing a reply when its worker dies instead of resuming generation is honest and simple; the cost is a "Regenerate" click on a rare failure.
• Routing by conversation for KV-cache hits saves seconds of prefill per turn; it skews load, so fall back to least-loaded when a replica is hot and accept the cache miss.
• Summarizing old turns keeps prompts in budget and cost bounded, but loses detail; retrieval of old turns gets some of it back at the price of an embedding index per conversation.

Mid-level (L4)

• SSE over HTTP for streaming with a correct reason against polling; a stop button that actually stops generation.
• Persists the finished message and explains what the client does on reconnect, even if the first answer is "reload the message".
• Knows the context window is finite and proposes truncation or a summary.

Senior (L5)

• Decouples generation from the connection with a per-reply log and sequence numbers; resume via Last-Event-ID, exactly-once rendering by dropping seq <= last.
• Splits the API into start and stream, makes the POST idempotent, and handles abandoned streams and proxy buffering.
• Works the numbers: concurrent streams, events per second with batching, Redis memory.

Staff (L6+)

• Ties long conversations to GPU cost: prefill seconds per turn, KV-cache size per conversation, routing by conversation, and why summaries should be rare and large.
• Is precise about failure: what is durable when, what happens when a worker or a Redis shard dies, and chooses honest failure over pretend failover.
• Covers multi-region placement and the operational edges (reconnect storms after a deploy, chaos-testing resume).