Design a stock trading platform

Users place market and limit orders; match them, update balances and positions exactly once, and stream prices in real time.

We are building a retail brokerage (Robinhood-style): users place market and limit orders, the broker holds buying power, routes each order to an exchange that does the matching, applies fills to cash and positions exactly once, and streams live quotes to the app.

Functional requirements

• Place market and limit orders (buy/sell, day or GTC), with an idempotency key from the client.
• Cancel an open order; cancels race with fills and the exchange decides the winner.
• Order status and history: pending → accepted → partially_filled → filled, or cancelled / rejected / expired.
• Account view: cash, buying power, positions with average cost, realized and unrealized P&L.
• Real-time quotes and order updates pushed to clients.
• Clarify up front: US equities only, cash accounts (no margin or short selling), whole and fractional shares, regular trading hours plus pre-market. Matching happens at the exchange; we are not building an exchange order book.

Non-functional requirements

• Correctness first: no fill is applied twice or lost; cash and positions always reconcile with the exchange and the clearing firm.
• A user can never spend the same dollar twice: buying power is held before an order leaves the broker.
• Order placement p99 under 100 ms broker-side (excluding exchange time); quote fan-out under 200 ms from exchange to phone.
• 99.99% availability during market hours; recover in-flight orders after any crash without user action.
• Full audit trail kept 7 years (SEC 17a-4, FINRA CAT reporting by T+1 8am).

Back-of-the-envelope estimation

• 20M accounts, 3M daily active traders, 10M orders/day. Average 430 orders/s over 6.5 hours, but market open (9:30–9:35) runs 20–30× average: plan for 15k orders/s peak.
• Each order produces about 4 events (new, ack, 1–2 fills) → 40M events/day × 1 KB ≈ 40 GB/day of order events; 15 TB/year before compression.
• Quotes: ~10k actively traded symbols, market data feed peaks at 1M+ updates/s. Clients don't need every tick: conflate to 4 updates/s per symbol.
• 2M concurrent app sessions × ~20 watched symbols → 40M subscriptions; at 4 Hz worst case that is too much to push raw, so conflate per connection and only push symbols on screen.
• Positions: 20M accounts × ~8 holdings × 100 B ≈ 16 GB. Fits a sharded relational store comfortably.

Clients use REST for commands and a WebSocket for streams. The broker talks FIX 4.4 to exchanges/market makers.

POST /v1/orders
Headers: Idempotency-Key: 6f1c...  (client UUID per order intent)
{ "account_id": "a_91", "symbol": "AAPL", "side": "buy", "type": "limit",
  "qty": "10", "limit_price": "187.50", "time_in_force": "day" }
→ 202 { "order_id": "o_7Q2", "status": "pending", "held_amount": "1875.00" }
→ 422 { "code": "insufficient_buying_power" } | 409 if key reused with a different body

DELETE /v1/orders/{order_id}
→ 202 { "order_id": "o_7Q2", "status": "pending_cancel" }   // final state arrives on the stream

GET /v1/orders/{order_id}
→ { "status": "partially_filled", "filled_qty": "4", "avg_fill_price": "187.42",
    "fills": [{ "exec_id": "NSDQ-88121", "qty": "4", "price": "187.42", "ts": "..." }] }

GET /v1/accounts/{id}/positions   GET /v1/accounts/{id}/balances   GET /v1/orders?status=open&cursor=...

WS /v1/stream
→ { "op": "subscribe", "channels": ["quotes:AAPL", "orders:a_91"] }
← { "ch": "quotes:AAPL", "bid": "187.48", "ask": "187.51", "last": "187.50", "seq": 9921 }
← { "ch": "orders:a_91", "order_id": "o_7Q2", "status": "filled", "seq": 44 }

Recovery contract: if a POST times out, the client retries with the same Idempotency-Key and gets the original order back, never a second order. Stream messages carry a per-channel seq; on reconnect the client sends last_seq and the server replays order events from the log (quotes are just resent as a snapshot).

Ledger-of-record is a sharded PostgreSQL (or Spanner/CockroachDB) cluster, partitioned by account_id, so an order, its hold, its fills and the account's balances update in one local transaction.

Kafka topics: orders.commands (partitioned by account), exec.reports (by order_id), order.events (by account), marketdata.quotes (by symbol). The full event stream is copied to WORM object storage for the audit trail. Quotes live only in memory (latest per symbol in Redis for snapshots).

flowchart LR
  App[Mobile and web] --> GW[API gateway]
  GW --> OS[Order service]
  OS --> DB[(Accounts and orders DB)]
  OS --> OB[Outbox relay]
  OB --> K[Kafka]
  K --> RT[Order router]
  RT -->|FIX| EX[Exchanges and market makers]
  EX -->|execution reports| RT
  RT --> K
  K --> FP[Fill processor]
  FP --> DB
  K --> PS[Push service]
  MD[Market data feed] --> QS[Quote service]
  QS --> PS
  PS -->|WebSocket| App
  K --> AU[Audit archive]

1. Place with hold. The order service checks the idempotency key, then in one transaction on the account's shard: computes the hold (limit price × qty, or last price × qty × 1.05 collar for market buys; shares for sells), checks buying power, inserts the order as pending, increments held, writes an outbox row. Returns 202.
2. Exchange submission. The outbox relay publishes to orders.commands. The order router runs pre-trade risk checks (price collar, fat-finger size, restricted symbols, market hours), picks a venue via smart order routing, sends a FIX NewOrderSingle with ClOrdID = order_id, and records submitted.
3. Fill processing. Each FIX ExecutionReport is written to exec.reports keyed by order_id. The fill processor, in one transaction: inserts the execution (dedupe on exec_id), updates filled_qty/avg_price/status, debits cash and releases the matching portion of the hold, updates the position and average cost, writes ledger entries and an outbox event. Duplicate reports hit the unique key and are skipped.
4. Cancel. The order service marks pending_cancel and emits a cancel command; the router sends OrderCancelRequest. The exchange replies either "cancelled" (release remaining hold) or "too late to cancel" (the fill already happened and is applied normally).
5. Streaming. The quote service normalizes the feed, keeps the latest quote per symbol, and conflates to 4 Hz. The push service holds WebSockets, subscribes to the symbols its clients watch, and forwards order events for its connected accounts.

Order lifecycle state machine (transitions only move forward; version enforces it):

pending → submitted → accepted → partially_filled → filled
   │          │           │              │
   └→ rejected └→ rejected └→ pending_cancel → cancelled (remaining qty) / filled (lost race)
accepted / partially_filled → expired (day order at close)

Exactly-once fills. The exchange delivers at-least-once (FIX resends after reconnect, our consumers can redeliver after a crash). We don't rely on Kafka transactions for correctness; we make the apply step idempotent. The executions row with PK (venue, exec_id) is inserted in the same DB transaction as the cash, hold, position and ledger changes, so a fill is either fully applied or not at all, and a replay is a no-op. FIX sequence numbers are persisted per session; on reconnect the router asks for a resend from the last stored number (ResendRequest), so gaps are closed rather than lost. An end-of-day reconciliation compares our executions with the clearing firm's drop copy and alerts on any difference.

Buying-power holds and concurrency. Two concurrent orders on one account must not both pass the check. Options:

Market buys hold at a collar (last ask × 1.05); if the fill would exceed it, the router sends the order as a marketable limit at the collar price so the hold always covers it. On a partial fill we release hold × filled_qty / qty − actual cost difference immediately, so the user sees buying power return.

Partial fills and positions. Each fill updates incrementally: new_avg = (old_qty × old_avg + fill_qty × fill_price) / (old_qty + fill_qty) for buys; sells reduce qty, keep avg_cost, and book realized P&L = fill_qty × (fill_price − avg_cost). Use decimal types (NUMERIC(20,8)), never floats. Fills for one order are applied in exec-report order because exec.reports is partitioned by order_id; a fill that arrives after a cancel ack (rare out-of-order venue behavior) is still applied, since the exchange is the source of truth, and the cancel then covers only the remainder.

Market-open bursts. 15k orders/s in the first minutes, plus pre-market queued GTC orders all releasing at 9:30. The order service is stateless and scaled up on a schedule before 9:25. The DB is sharded by account (64 shards, each ~250 writes/s at peak, well within limits). Kafka absorbs router backpressure: if a venue slows, commands queue rather than failing the user's request. Per-venue FIX sessions are the real bottleneck (each handles a few thousand msg/s), so run several sessions per venue and shard orders across them by order_id. Shed load in order of importance: quote frequency first, history reads second, order placement last.

Exchange failover and circuit breaking. The router tracks per-venue ack latency and reject rate. If acks stop for 2 s or rejects spike, a circuit breaker opens and the smart order router sends new orders elsewhere. Orders already at the dead venue are left as accepted with status unknown; we never resubmit them to another venue, because the original may still fill and the user would end up with double the shares. After reconnect, the router sends OrderStatusRequest for each open order and applies what comes back. Market-wide halts (LULD, circuit breakers at −7/−13/−20%) are surfaced to users, and new orders for halted symbols are rejected.

Audit and compliance. Every state change is an event in Kafka, archived to WORM storage (S3 Object Lock) with the FIX messages themselves. A nightly job builds the FINRA CAT report from these events. The ledger_entries table is append-only; corrections are new reversing entries, never updates.

• Holding buying power at placement makes double spending impossible, but over-holds for market orders (5% collar) and temporarily lowers what the user can spend.
• Sharding by account keeps every money movement in a single local transaction; cross-account views (firm-wide exposure per symbol) need a separate streaming aggregate and are seconds behind.
• Idempotent apply with DB unique keys is simple and correct, but every fill is a DB transaction; Kafka exactly-once would cut writes but ties correctness to Kafka config.
• Outbox plus Kafka adds 10–50 ms between "order accepted" and "order at the exchange", in return for never losing an order between the DB and the router.
• Conflating quotes to 4 Hz cuts fan-out bandwidth by 100× or more; users see the right price within 250 ms but not every tick (pro traders would need a paid full feed).
• Never resubmitting an order with unknown status leaves a user stuck for a few seconds during a venue outage, instead of risking a duplicate execution.
• Routing to external venues instead of matching internally avoids running an exchange, but makes us depend on venue uptime and payment-for-order-flow economics.

Mid-level (L4)

• Clear order API with idempotency key; order states and a reasonable schema with orders, fills, balances, positions.
• Checks buying power before sending an order and sees the race between two concurrent orders.
• Separates order placement from quote streaming and uses WebSockets for pushes.

Senior (L5)

• Holds funds atomically (conditional update or row lock) and releases them on fill, cancel and reject.
• Makes fill processing idempotent with an execution-ID unique key inside the same transaction as balance and position updates; explains why at-least-once delivery plus dedupe gives exactly-once effects.
• Handles partial fills, average cost and the cancel/fill race with an explicit state machine.
• Sizes the market-open burst and quote fan-out, and conflates quotes.

Staff (L6+)

• Designs recovery end to end: transactional outbox, FIX sequence-number resend, order status queries after reconnect, and never resubmitting orders with unknown status.
• Brings in reconciliation with the clearing firm, an append-only double-entry ledger, and regulatory audit (CAT, WORM retention).
• Discusses venue circuit breaking, load shedding order, market halts, and the cost of the collar and sharding choices.