Design an online chess game

Match players of similar rating, play live games with clocks, handle disconnects and reconnects, and keep game history and ratings.

We are building a chess.com-style service: players queue for a rated game at a time control, get paired with someone of similar rating, play live with server-owned clocks, survive dropped connections, and end with a durable result, PGN history and updated ratings.

Functional requirements

• Queue for a game by time control (bullet 1+0, blitz 3+2 / 5+0, rapid 10+0) and rated/casual; cancel the queue.
• Pair players of similar rating, widening the acceptable gap the longer they wait.
• Play a live game: submit moves, see the opponent's moves within a few hundred ms, with per-player clocks and increment.
• Resign, offer/accept draw, claim threefold repetition / 50-move rule; automatic checkmate, stalemate and timeout detection.
• Reconnect to an in-progress game from any device and resume with the correct board and clocks.
• Abort before both sides have moved; a no-show loses nothing and costs no rating.
• Game history (PGN, move times), rating per time control, rematch, spectators.

Non-functional requirements

• Exactly one committed move per ply: no lost, duplicated or out-of-order moves, even with retries and double clicks.
• Clocks are fair: the server decides elapsed time; client clocks are display only. Network lag should not cost a player more than ~100 ms per move (lag compensation).
• Move commit to opponent delivery p99 under 200 ms in-region.
• A game survives a server crash: the authoritative state is durable after every move.
• Matchmaking wait under 10 s for popular pools at peak.
• Fair-play (engine detection) never sits on the move path.

Back-of-the-envelope estimation

• 10M DAU, 1M concurrent players at peak → ~500k live games.
• Average blitz move every ~4 s per game → 500k / 4 ≈ 125k moves/s at peak.
• Each move: one validate + one durable write + two pushes (opponent, self ack) + spectator fanout.
• Game record: ~80 plies × ~20 B (SAN + clock) ≈ 2 KB PGN; 20M games/day → 40 GB/day, ~15 TB/year.
• Live game state: FEN (~90 B) + clocks + move list ≈ 3 KB → 500k × 3 KB = 1.5 GB in memory.
• Matchmaking: ~30k queue entries per minute per popular pool; tiny state, but high churn.

Principles: the client never decides truth. Every mutating call carries an idempotency key or an expected ply; the server answers with the canonical state. HTTP/RPC commits moves; the websocket only pushes.

POST   /v1/matchmaking/tickets         {time_control: "3+2", rated: true}
       → 202 {ticket_id, status: "searching"}
DELETE /v1/matchmaking/tickets/{id}    → 200 {status: "cancelled"} | 409 {status: "matched", game_id}

GET    /v1/games/{id}                  → {game_id, white, black, fen, ply, moves[],
                                          clocks: {white_ms, black_ms}, turn, turn_started_at,
                                          status, result, server_time}
POST   /v1/games/{id}/moves            {ply: 23, uci: "e2e4", client_move_id}
       → 200 {ply: 24, fen, clocks, status} | 409 {code: "stale_ply", game} | 422 {code: "illegal"}
POST   /v1/games/{id}/actions          {type: "resign"|"offer_draw"|"accept_draw"|"abort"|"claim_draw", ply}
GET    /v1/games/{id}/events?after=23  → [{seq, type, ...}]   // catch-up after reconnect

WS     /v1/live?game_id=...            server → client: move_committed, clock_sync,
                                       draw_offered, opponent_disconnected, game_over

Why not websocket-only moves. A move sent over a socket that drops has unknown fate; the client must retry and the server must dedupe anyway. Making the commit an idempotent request keyed by (game_id, ply) gives a clear ack, and the socket only does fanout. In practice the same request can ride the open websocket as an RPC frame with a request id to avoid a TLS handshake, but the semantics are the same: expected ply in, canonical state out.

• Live game state (Redis, or in memory of the owning game server, replicated): key game:{id} → {fen, ply, moves, white_ms, black_ms, turn_started_at, status, draw_offer, version}. Hot, small, mutated every move.
• Move log (durable, append-only): game_moves(game_id, ply, uci, san, clock_after_ms, server_ts), PK (game_id, ply). The unique PK is what enforces "exactly one move per ply". Stored in a partitioned SQL table or Cassandra/DynamoDB keyed by game_id.
• Games (Postgres/MySQL, sharded by game_id): games(id, white_id, black_id, time_control, rated, status, result, termination, started_at, ended_at, white_rating_before, black_rating_before, pgn). Indexes on (white_id, ended_at) and (black_id, ended_at) for history; or a separate player_games(player_id, ended_at, game_id) table sharded by player.
• Ratings (SQL): ratings(player_id, pool, rating, rd, volatility, games, updated_at), PK (player_id, pool). Glicko-2 needs RD and volatility.
• Matchmaking (Redis): one sorted set per pool mm:{pool} scored by rating, plus ticket:{id} hash with player_id, rating, rd, enqueued_at, status. player:{id}:active holds the current ticket or game to block double-queueing.
• Archive: finished PGNs to object storage in daily batches for analytics and fair-play.

SQL for games and ratings because results and rating updates need transactions; Redis for queue and live state because they churn and need sub-ms access.

flowchart LR
  C[Client] --> GW[API gateway]
  C <--> WS[Websocket edge]
  GW --> MM[Matchmaker]
  MM --> Q[(Redis queues)]
  MM --> GS[Game service]
  GW --> GS
  GS --> LS[(Live state)]
  GS --> ML[(Move log)]
  GS --> TQ[Timeout scheduler]
  GS --> BUS[Event bus]
  BUS --> WS
  BUS --> RT[Rating service]
  BUS --> FP[Fair play]
  RT --> DB[(Games and ratings DB)]
  GS --> DB

1. Queue and match. The client creates a ticket. The matchmaker, every 250 ms per pool, scans the sorted set and pairs tickets whose rating windows overlap; windows widen with wait time. It atomically claims both tickets and asks the game service to create the game, which writes the games row (status created) and the live state. Both players get matched {game_id} via the socket.
2. Join and start clocks. Each client opens the live socket and GETs the game. White's clock does not run until White makes the first move; Black's first-move window opens after that. A no-show within 30 s (bullet 15 s) aborts the game with no rating change.
3. Move. The client posts {ply, uci, client_move_id}. The game service, as single writer for that game, checks the ply, validates legality, computes elapsed time from turn_started_at, subtracts it from the mover's clock, adds increment, appends (game_id, ply) to the move log, updates live state, reschedules the timeout for the opponent and publishes move_committed. The socket edge pushes it to both players and spectators.
4. Reconnect, timeout, finish. A reconnecting client fetches the canonical game plus events after its last ply. The timeout scheduler fires when the side to move's clock would hit zero; the game service rechecks and ends the game (or declares a draw if the opponent has insufficient material). Game end writes the result and PGN in one transaction and emits game_over; the rating service updates both ratings from that event.

Matchmaking tickets, widening, and safe reservation.

• Pools are (time_control, rated). Each ticket's acceptable range starts at ±50 and widens by ~25 per 2 s waited, capped at ±400 (or use rating ± k × RD so uncertain new players get wider windows).
• A matcher worker owns each pool (leader lease), so pairing is not raced between workers. For each ticket in wait order, it looks up the sorted set within its window and picks the nearest rating whose own window also includes it.
• Reservation is atomic: a Lua script moves both tickets from searching to matched only if both are still searching, and sets player:{id}:active. Cancellation uses the same script, so a cancel and a match cannot both succeed; the loser of the race sees 409 with the game id.
• If game creation fails after reservation, the matcher returns both tickets to searching with their original enqueued_at. Game creation is idempotent on a match_id.
• Avoid immediate rematch with the same opponent unless both ask; keep a short recent-opponent set.

Authoritative move commit, legality, and exactly one next ply.

• Each live game has one owner: games are hashed to game-server shards, and a lease (etcd/Redis, 10 s TTL, fenced by epoch) names the owning node. All moves for a game route to it, so moves are processed serially in memory with no locking.
• Commit order: check ply == current_ply → check client_move_id (a retry of the last committed move returns the stored result) → legality with a bitboard move generator (~microseconds) → compute clocks → durable append with conditional write on (game_id, ply) and the owner's epoch → update memory → publish.
• The durable write is the commit point. If the node dies before publishing, the new owner reloads from the move log and republishes; clients reconcile with GET.
• A stale owner (lost lease) fails the conditional write because its epoch is old, so two nodes can never commit different moves at the same ply.
• Premoves are stored client-side and sent the instant the opponent's move arrives; the server treats them as ordinary moves.

Clock truth, increment, timeout, and reconnect recovery.

• The server stores remaining_ms per side and turn_started_at (server monotonic time). Elapsed for a move = commit_time − turn_started_at − lag_credit. Clients display a local countdown re-anchored on every clock_sync.
• Lag compensation: measure RTT with socket pings; credit up to min(RTT/2, 100 ms) per move, with a per-game quota (for example 1 s total in bullet) so it cannot be abused.
• Timeout: on every commit, schedule deadline = now + remaining_ms for the side to move in a timer wheel on the owning node, backed by a Redis sorted set timeouts scored by deadline so a failover node picks them up. When it fires, recheck under the game's serial queue; a move that committed first wins.
• Flag is a loss only if the opponent has mating material; otherwise draw.
• Disconnects do not pause the clock in rated play. Show "opponent disconnected" after 5 s; if they don't return within a grace window (30–60 s, or their clock running out, whichever first), the remaining player may claim the win.
• Reconnect: client sends last seen ply and event seq; server replies with full state if behind by more than a few events, else the missed events. The board is always rebuilt from canonical state, never from cached client guesses.

Abort, game end, ratings, and rematch.

• Before both sides have moved, either may abort; no rating change, no history noise beyond an "aborted" row. Repeated aborts or no-shows trigger a short queue cooldown.
• Game end is one transaction: games.status = finished, result, termination, PGN. An outbox row in the same transaction drives game_over to the bus, so the rating update can't be lost or doubled.
• The rating service applies Glicko-2 per game, keyed by game_id (idempotent), updating both players in one transaction. Store before/after ratings on the game row for display.
• Rematch creates a new game with colors swapped if both accept within 20 s; tournament games use the same game service with a tournament_id and report results to the tournament service through the same event.

Spectators vs player-critical delivery.

Fair play reads the finished-game event stream and per-move timings off the bus; results go to a review queue and, if confirmed, adjust ratings retroactively. Nothing on the move path waits for it.

• Single owner per game makes ordering trivial and fast, but needs leases, fencing and failover; a node loss pauses its games for a few seconds until a new owner loads them.
• Durable write per move costs a few ms and ~125k writes/s, but makes every committed move survive crashes. Batching writes would be cheaper and lose the last move on failure.
• HTTP/RPC commit with expected ply adds explicit acks and idempotency; websocket-only fire-and-forget is simpler and faster but leaves move fate unknown on a drop.
• Server-owned clocks are fair and cheat-proof, but punish high-latency players; capped lag credit narrows that without letting clients claim time.
• Widening windows shorten waits at the cost of less even games for players at the rating extremes.
• Not pausing clocks on disconnect keeps the game fair against deliberate disconnects but is harsh on genuine network drops.
• Async fair play keeps moves fast but means a cheater's wins stand until review catches up.

Mid-level (L4)

• Clear requirements, a working queue-to-game flow, a game table and move table, websocket push of moves.
• Server validates legality and keeps clocks; knows the client clock is display only.
• Basic rating update (Elo) after the game ends.

Senior (L5)

• Atomic ticket reservation, widening rating windows, cancel/match race handled.
• Expected-ply commits with idempotent retries and a unique (game_id, ply) key; explains why websocket-only moves are risky.
• Timeout scheduling that survives failover, reconnect via canonical state plus event catch-up, abort/no-show rules, idempotent rating updates via outbox.

Staff (L6+)

• Single-writer ownership with leases and epoch fencing; reasons about exactly what happens when an owner dies mid-commit.
• Lag compensation with bounded credit, disconnect policy as a fairness decision, and clock correctness at scale.
• Separates player-critical delivery from spectator fanout, keeps fair play and tournaments off the hot path, and sizes the system (moves/s, live state, storage growth).