Design real-time comments on a design canvas

Source: https://www.hack2hire.com/question-bank/questions/system-design/6a0cab29567356be0f453385
Company: Figma
Difficulty: Hard

Problem statement

Let people pin comments to points and objects on a shared design canvas, reply in threads, and see each other's comments live as the design changes underneath them.

Requirements

We are building the comment layer of a multiplayer design tool: people drop pins on points or objects of an infinite canvas, discuss them in threads, and everyone with the file open sees new comments, replies and resolves within a second, while the design keeps moving underneath.

Functional requirements

• Create a thread by pinning it to a canvas point, or to an object (frame, layer) so the pin follows that object when it moves or resizes.
• Reply, edit, delete, react, @mention, and resolve/reopen threads.
• Everyone viewing the file sees thread and reply changes live; viewers also see who is typing in a thread.
• List comments for a file, filtered by page, open/resolved, and "mentions me"; load only what is near the viewport on dense files.
• After a dropped connection, catch up exactly on what was missed, with no duplicates and no full reload.
• Notify mentioned and participating users who are not online (email, in-app inbox).

Out of scope: the design document's own sync (it already has a multiplayer engine; we read object positions from it), and permissions beyond "can view file ⇒ can read comments; can comment ⇒ can write".

Non-functional requirements

• Live delivery p99 under 500 ms from write to every connected viewer of the file.
• Writes are durable once acknowledged; a comment never silently disappears.
• Per-file ordering: all viewers apply events for one file in the same order.
• Hot files (a design review with 500 viewers, a company all-hands file with 5k) must not slow down normal files.
• Reads dominate: opening a file loads its comments; that load must be under 200 ms p95.

Back-of-the-envelope estimation

• 10M DAU, 2M concurrent sessions at peak, each with one file open.
• 20M comments/day ≈ 230 writes/s average, ~1.5k/s peak. Small, so the write path is not the hard part.
• Fanout: average 3 viewers per open file, but the tail matters: top 0.1% of files have 100–5,000 viewers. One reply on a 5k-viewer file is 5k pushes.
• Total pushes ≈ 1.5k writes/s × ~10 avg weighted viewers ≈ 15k msgs/s, bursting to 100k+/s during big reviews. Typing indicators are 10× that if not throttled.
• Storage: ~1 KB per comment with metadata → 20 GB/day, ~7 TB/year. Typical file has under 200 threads; the 99.9th percentile has 20k+.
• WebSocket gateways: 2M connections at ~50k per node ≈ 40–60 nodes.

API design

REST for durable reads and writes, one WebSocket per open file for live events. Every write returns the per-file sequence number it was assigned.

POST /v1/files/{file_id}/threads
  {client_id, page_id, anchor: {type: "point"|"object", x, y, node_id?, offset_x?, offset_y?},
   body, mentions: [user_id]}
  → 201 {thread_id, comment_id, seq, created_at}

POST /v1/threads/{thread_id}/comments      {client_id, body, mentions}  → 201 {comment_id, seq}
PATCH /v1/comments/{comment_id}            {body, expected_version}     → 200 {seq} | 409
POST /v1/threads/{thread_id}/resolve       {}                           → 200 {seq}
POST /v1/threads/{thread_id}/anchor        {anchor}                     → 200 {seq}   (pin dragged)
DELETE /v1/comments/{comment_id}                                         → 200 {seq}

GET /v1/files/{file_id}/threads?page_id=&status=open&bbox=x0,y0,x1,y1&cursor=
  → {threads: [{thread_id, anchor, status, reply_count, last_activity, preview}], seq, next_cursor}
GET /v1/threads/{thread_id}/comments?cursor=  → {comments: [...], next_cursor}
GET /v1/files/{file_id}/events?after_seq=N&limit=500 → {events: [...], latest_seq} | 410 if too old

WebSocket wss://rt.example.com/files/{file_id}?after_seq=N:

• Server → client: {seq, type: "thread.created"|"comment.added"|"comment.edited"|"thread.resolved"|"anchor.moved", thread_id, payload}; ephemeral {type: "typing", thread_id, user_id} without a seq.
• Client → server: typing, ack {seq} (optional, for lag metrics). Writes go through REST, not the socket, so they get normal auth, rate limits and retries.

client_id is a client-generated UUID, used as an idempotency key so a retried POST after a timeout does not create a second comment.

Data model

Comments are relational, small, and queried by file: Postgres (or Vitess/MySQL) sharded by file_id, so one file's threads, comments and event log live on one shard and get one transaction.

Every write is one transaction: update threads/comments, last_seq = last_seq + 1, insert file_events(seq). That gives a gap-free, per-file total order without a distributed sequencer. A CDC stream (Debezium or the outbox table read by a relay) publishes file_events to Kafka partitioned by file_id.

Hot path state in Redis: presence:{file_id} (set of gateway nodes holding subscribers, TTL 60 s), typing:{thread_id} (TTL 5 s).

High-level design

flowchart LR
  C[Client canvas] -->|REST writes and loads| API[Comment API]
  C <-->|WebSocket| GW[WS gateways]
  API --> DB[(Postgres sharded by file)]
  DB -->|outbox CDC| K[Kafka file events]
  K --> FO[Fanout workers]
  FO -->|"lookup file to gateways"| R[(Redis subscriptions)]
  FO --> GW
  GW --> R
  K --> N[Notification service]
  N --> E[Email and inbox]
  API -->|"resolve object positions"| DOC[Multiplayer doc service]

1. Place a pin. The client shows the pin immediately (optimistic, keyed by client_id). It POSTs the anchor in canvas coordinates plus node_id if dropped on an object. The API checks permission, writes the thread, the first comment, and file_events(seq=1042) in one transaction, and returns seq.
2. The outbox relay publishes the event to Kafka partition hash(file_id). A fanout worker reads it, looks up which gateway nodes hold subscribers for file_id in Redis, and sends one message per gateway node (not per user).
3. Each gateway pushes the event to its local sockets for that file. Clients apply events in seq order; the author's client sees its own client_id and swaps the optimistic pin for the real one.
4. The notification service consumes the same topic and emails mentioned users who have no live socket on the file after a 2-minute delay (batched, deduped).
5. Open a file. Client calls GET threads?page_id&bbox for the visible area and gets seq = S with the snapshot; it then opens the socket with after_seq=S, so nothing between the snapshot and the subscription is lost.
6. Reconnect after a drop. The client reconnects with its last applied seq. The gateway replays events after it from a Redis ring buffer (last 1,000 events per hot file) or from file_events; if the gap is older than retention, it returns 410 and the client refetches the snapshot.

Deep dives

Anchoring in canvas space. Pins must survive zoom, pan, and edits to the design.

• Store coordinates in canvas (world) space, never screen space; the client converts using the viewport transform, so zoom never touches the server.
• A point anchor is (page_id, x, y). An object anchor is (node_id, offset_x, offset_y) where the offset is normalized to the object's bounds (0..1), so the pin stays on "the top-right of the button" when the button is resized.
• The client resolves node_id against its local copy of the document, which the multiplayer engine already keeps live, so moving a frame moves its pins with zero comment-service traffic.
• If the node is deleted, fall back to the last known absolute position (stored on every anchor update) and mark the thread "detached" so the UI can show that the target is gone. Nested nodes: anchor to the deepest node clicked, and fall back up the parent chain recorded at pin time.
• Copying a frame to a new page does not copy comments; duplicating a file can, by remapping node ids.

Fanout for hot files. The naive design, one pub/sub message per subscriber, breaks at 5k viewers.

Pick the third. Gateways register file_id → node in Redis when the first local socket subscribes and remove it when the last leaves. For the top files, also batch: coalesce events per file into 50 ms frames, and throttle typing indicators to 1 per user per 3 s, sampled to at most 10 typers shown. Use consistent hashing of file_id to gateway nodes when the load balancer can route by file, so a hot file's viewers concentrate on a few nodes; spread across more nodes only when a file passes ~2k viewers on one node.

Reconnect and catch-up with version cursors. Mobile and laptop sleep make drops common (~5% of sessions per hour).

• The per-file seq is the cursor. The client stores its last applied seq; the gateway replays seq > N.
• Replay sources in order: Redis ring buffer (events:{file_id}, capped list of 1,000, TTL 1 h) → file_events table (30 days) → 410 Gone → full snapshot refetch.
• Clients ignore events with seq <= last_applied (duplicates are normal during replay overlap) and buffer out-of-order ones until the gap fills, then request the missing range.
• Jittered exponential backoff (1 s, 2 s, 4 s … 30 s cap) prevents a gateway deploy from causing a reconnect storm; gateways drain sockets over 5 minutes on deploy.
• Ordering only matters per file, which is why one shard and one Kafka partition per file are enough. There is no global order to maintain.

Viewport-aware loading for dense files. A file with 20k threads cannot ship them all on open.

• Bucket each thread into a tile grid (tile = floor(x / 4096)) per page; GET threads?bbox= reads only intersecting tiles, then the client prefetches neighbors while the user pans.
• When zoomed out, the server returns clusters (count per tile) instead of pins; the client expands a cluster on zoom.
• Live events for all threads still arrive (they are small), but bodies are fetched lazily when a thread is opened; events carry a 140-char preview.
• Object-anchored pins whose object moved across tiles get their tile updated when the client reports anchor.moved after a significant move (more than one tile), debounced to 2 s.

Edits, deletes and lifecycle. Comment edits use expected_version for optimistic concurrency (409 on conflict; only the author edits, so conflicts are rare). Deletes are soft (deleted_at) and broadcast as tombstones so replaying clients remove them; a nightly job hard-deletes after 30 days. Resolved threads are hidden by default but kept; deleting a file cascades through an async job, not a transaction. Moderation and legal holds act on comments directly and emit normal events so connected clients update.

Trade-offs

• Writes over REST and reads over WebSocket give a simple, retryable, rate-limited write path, but cost an extra round trip compared to writing on the socket.
• A per-file sequence in the same shard gives strict order and cheap catch-up, but it serializes writes per file; at a few writes/s per file that is fine, and it would not be for character-level document edits.
• Resolving object anchors on the client avoids coupling the comment service to the document model, but the server cannot answer "where is this pin" for email previews or exports without asking the document service.
• Gateway-level fanout keeps hot files cheap but adds the Redis subscription map, which can be briefly stale; the cost is a few dropped live events, covered by seq gap detection.
• Optimistic UI makes pinning feel instant but requires idempotency keys and a visible "failed to send" state.
• Clustering when zoomed out hides individual pins, trading discoverability for render speed.

Interviewer expectations

Mid-level (L4)

• Clean thread/comment schema, REST endpoints, and a WebSocket channel per file.
• Stores pin positions in canvas coordinates, not pixels.
• Knows a dropped socket needs some resync, even if it is "refetch everything".

Senior (L5)

• Object anchors with normalized offsets that follow moves and resizes, plus a fallback when the object is deleted.
• Per-file sequence numbers, snapshot-then-subscribe without a gap, and replay on reconnect with dedup.
• Separates durable writes from fanout through an outbox and a log; idempotent creates.
• Sizes the hot-file problem and throttles typing indicators.

Staff (L6+)

• Gateway-level fanout and routing by file, with numbers for the 5k-viewer case and reconnect storms on deploy.
• Viewport tiling and clustering for 20k-thread files, and a clear line on what the comment service owns versus the document engine.
• Operational view: lag from write to delivery, seq-gap rate, reconnect rate, and replay misses (410s) as the health signals, with SLOs and alerts on them.
• Discusses retention, deletes as tombstones, and how moderation or legal holds flow through the same event path.