Design a payment system

Accept card payments for merchants: authorize, capture, settle with card networks, and reconcile, never charging twice or losing money.

We are building a card payment processor that sits between merchants (terminals and online checkouts) and the card networks: it places authorization holds in real time, accepts captures (final charges, often with a tip added), settles them with the networks in a nightly batch, and reconciles what the networks paid against what we recorded. The one rule above all others: a customer is never charged twice and no money goes missing.

Functional requirements

• Authorize (hold): reserve an amount on a card and return approve/decline within the checkout window.
• Capture (charge): turn a hold into a charge for the final amount, which may differ from the hold (tip, partial shipment). One capture per hold by default; partial captures where the merchant enables them.
• Void / expire: release a hold the merchant cancels, and release holds that are never captured.
• Refund: return all or part of a settled charge.
• Settle: submit all captures from the day to each network in a batch, and track which ones the network accepted.
• Reconcile: match network settlement files and bank deposits against our ledger; flag every mismatch.
• Status lookup: merchant can query any payment by id or by its idempotency key.

Out of scope: fraud scoring models, disputes/chargebacks workflow (mentioned only where they touch the ledger), payouts to merchant bank accounts beyond the ledger entry.

Non-functional requirements

• Exactly-once effect: retries from terminals, our own services, or the networks never create a second hold, charge, or settlement line.
• No lost money: every accepted capture is durable before we say "accepted"; every cent is in a double-entry ledger that balances.
• Latency: authorization p99 under 1.5 s end to end, of which our own work is under 150 ms (the rest is the issuer).
• Availability: 99.99% for authorize and capture. Settlement is batch and can be retried, so it tolerates hours of delay but never a missed day.
• Auditability: immutable history of every state change, kept 7+ years (PCI DSS, financial audit).
• Security: raw card numbers never leave a tokenization vault (PCI scope kept small).

Back-of-the-envelope estimation

• 100M payments/day ≈ 1,200/s average, 10,000/s peak (holiday evening, 8× average).
• Each payment produces ~3 writes on the hot path (hold, capture, ledger entries) and ~5 state rows over its life ≈ 500M rows/day.
• Payment row ~1 KB with history, ledger entries ~200 B × 4 → ~180 GB/day, ~65 TB/year before compression; partition by month and move cold months to cheaper storage.
• Settlement: 100M captures across ~4 networks per night; a network file of 25M lines at ~300 B ≈ 7.5 GB, which must be built and sent inside a ~2 hour window before the network cutoff.
• Idempotency keys: 100M/day kept 7 days ≈ 700M keys × 100 B ≈ 70 GB.

Merchants call a synchronous REST API; settlement and reconciliation are internal batch jobs. Every mutating call requires an Idempotency-Key header.

POST /v1/authorizations
Idempotency-Key: 6f1c...           # per merchant, client-generated
{ "merchant_id": "m_42", "amount": 5000, "currency": "USD",
  "card_token": "tok_9x", "entry_mode": "chip", "terminal_id": "t_7" }
→ 201 { "payment_id": "pay_123", "status": "authorized",
        "authorized_amount": 5000, "expires_at": "2026-10-19T23:59:59Z" }
→ 402 { "payment_id": "pay_123", "status": "declined", "decline_code": "insufficient_funds" }

POST /v1/payments/{payment_id}/capture
Idempotency-Key: a81b...
{ "amount": 5750, "tip_amount": 750 }
→ 202 { "payment_id": "pay_123", "status": "captured", "capture_id": "cap_88",
        "settlement_date": "2026-10-12" }

POST /v1/payments/{payment_id}/void        → 200 { "status": "voided" }
POST /v1/payments/{payment_id}/refunds     { "amount": 2000 } → 202 { "refund_id": "re_5", "status": "pending" }
GET  /v1/payments/{payment_id}             → full status + event history
GET  /v1/payments?idempotency_key=6f1c...  → same, for clients that lost the response

• Capture returns 202 Accepted, not "money moved": it means the capture is durably recorded and will be in tonight's batch. Funds move at settlement.
• Same key + same body → replay the stored response. Same key + different body → 422 idempotency_key_reused. Key still in flight → 409 request_in_progress with Retry-After: 1.
• Webhooks (payment.settled, payment.settlement_failed, refund.settled) tell merchants about batch outcomes.

Why batch settlement, not a network call per capture: networks clear funds once per day per merchant anyway; batch files carry interchange qualification data, are cheaper per item, and decouple merchant-facing availability from network availability. Capture only needs our own database to be up.

Store: a sharded relational database (PostgreSQL/Spanner-class) for payments and the ledger, because we need multi-row ACID transactions (state change + ledger entries + outbox row commit together) and strong uniqueness constraints. Shard key is merchant_id, so one payment and all its rows live on one shard.

Lifecycle states: authorized → captured → submitted → settled, with side exits declined, voided, expired, settlement_failed → (retry) submitted, and settled → partially_refunded / refunded. Transitions are checked in code against an allowed-transition table and applied with UPDATE ... WHERE payment_id = ? AND status = ? AND version = ?, so two racing captures cannot both win.

Core invariants: captured_amount ≤ authorized_amount × (1 + tip_tolerance); refunded_amount ≤ captured_amount; each capture appears in exactly one accepted batch line; the ledger balances per transaction and per day.

flowchart LR
  T[Terminal or checkout] --> GW[API gateway]
  GW --> PS[Payment service]
  PS --> V[Token vault]
  PS --> DB[(Payments and ledger DB)]
  PS --> NA[Network adapter]
  NA --> CN[Card networks]
  DB --> OB[Outbox relay]
  OB --> K[Event log]
  K --> WH[Webhook service]
  SCH[Settlement scheduler] --> DB
  SCH --> SF[Batch file builder]
  SF --> CN
  CN --> RF[Settlement and bank files]
  RF --> RC[Reconciliation service]
  RC --> DB
  SW[Expiry sweeper] --> DB

1. Authorize. Gateway authenticates the merchant and rate-limits. The payment service inserts the idempotency key as in_flight, creates the payment in pending_auth, detokenizes the card inside the vault boundary and sends an ISO 8583 authorization through the network adapter. On approve, it stores the auth code and network transaction id, moves to authorized, writes a ledger hold, and stores the response against the key, all in one transaction. On timeout it sends a reversal (see deep dives).
2. Capture. One local transaction: check state and amount limits, insert the capture with settlement_date = the current business day, move the payment to captured, write ledger entries (customer receivable / merchant payable), write an outbox event, mark the key done. Returns 202. No network call.
3. Nightly settlement. At each network's cutoff the scheduler freezes the set of captures for that network and date into a batch, the builder writes the file, signs and uploads it, and records the network's acknowledgment. Accepted lines move to submitted; rejected lines move to settlement_failed and are retried in the next batch or escalated.
4. Reconciliation. Next morning the network sends a clearing/settlement report and the bank shows a deposit. The reconciliation service matches report lines to captures, totals to the batch, and the deposit to net settlement (gross − interchange − fees). Matched captures move to settled; anything else becomes a recon_item for automated retry or a human.
5. Expiry. A sweeper releases holds past expires_at (typically 7 days, network-dependent) with a reversal so the cardholder's funds are freed.

Idempotency across hold, capture and settlement. Each step has its own dedupe key, because retries happen at every hop.

The in_flight key state handles concurrent duplicates: the second request sees in_flight and gets 409, instead of starting a second network call. A key stuck in_flight past 60 s is resolved by a recovery job that asks the network (or our own payment row) what happened, rather than by guessing.

Nightly batch settlement: cutoffs, partial failure, completion. Each network has its own cutoff (for example 22:00 ET) and each merchant a business-day timezone. The cutoff is a hard boundary on captures.created_at in that zone: a capture committed at 21:59:59.900 is in today's batch; at 22:00:00.100 it gets tomorrow's settlement_date, and the capture response already told the merchant which date. Batch building works in steps, each persisted so a crash resumes instead of restarting:

1. OPEN → FROZEN: in one transaction per shard, assign batch_id to all unbatched captures for that network and date. Shards run in parallel (100 shards × 1M rows each); the batch is frozen only when every shard reports done and per-shard counts and totals are recorded.
2. FROZEN → FILE_BUILT: stream rows to the file with a trailer of count and total; store checksum.
3. FILE_BUILT → SENT → ACKED: upload over SFTP or the network's API; poll for the acknowledgment. Retry the same file (same id) on failure; alert if not acked 60 minutes before the network's hard deadline.
4. ACKED → RECONCILED: once the clearing report arrives.

Partial failure: networks reject individual lines (bad data, expired auth). Those lines go to settlement_failed with the reason; fixable ones are re-queued into the next night's batch with a new line id but the same capture_id, so reconciliation still ties them together. Completion is tracked per batch and per day by a dashboard of counts and totals at each step; a day is "closed" only when every batch is reconciled.

Reconciliation: closing the trust gap. Three independent records must agree: our ledger, the network's clearing report, and the bank deposit. Matching is a three-way join by network_txn_id (fallback: auth code + amount + date). Outcomes:

Targets: >99.9% auto-match; every exception has an owner and age; unknown-at-network alerts within minutes of the file landing. Ledger totals per day are compared to network totals before any merchant payout runs, so a mismatch blocks paying out money we have not received.

Holds, tips and expiry. Restaurants authorize the bill and capture with tip. We allow capture up to authorized + 20% (a configurable tolerance by merchant category); above that the capture is rejected or needs an incremental authorization, which is a second network call that raises the hold. Partial capture releases the remainder with a partial reversal so the cardholder is not left with a stale hold. The sweeper scans (status, expires_at) hourly and reverses expired holds in batches of 1,000, rate-limited per network.

Downstream health and circuit breaking. The network adapter keeps a breaker per network and per issuer BIN range: if timeouts exceed 5% over 30 s, it opens, and new auths for that route return a fast issuer_unavailable decline (or go to stand-in processing where the network offers it) instead of piling up threads. Timed-out auths always get a reversal, queued durably and retried until acknowledged. Capture keeps working during a network outage because it touches only our database.

• Batch settlement over per-capture forwarding: gives isolation from network outages and lower cost; gives up same-second finality, so merchants see "captured" a day before "settled".
• Relational, sharded by merchant: gives ACID across payment, ledger and outbox; gives up cross-merchant queries on the hot path (those go to a warehouse fed by the event log) and makes very large merchants hot shards that need sub-sharding.
• Reversal on auth timeout instead of retry: may decline a payment the issuer actually approved (a lost sale), in exchange for never leaving a hidden duplicate hold.
• Outbox instead of publishing events directly: adds a relay and a few hundred ms of webhook delay, but events can never disagree with the database.
• Strict cutoff by commit time: simple and auditable; captures near the boundary slip a day, which merchants must accept.
• Tip tolerance: convenience for restaurants at the cost of a small, bounded over-capture risk the issuer may dispute.

Mid-level (L4)

• Separates authorize from capture and explains why holds exist.
• Uses idempotency keys on the merchant API and a unique constraint to enforce them.
• Draws a clear state machine and a relational schema with sensible indexes.

Senior (L5)

• Handles the auth-timeout case with reversals rather than retries, and dedupes at every hop, not just the API.
• Explains batch settlement with cutoffs, resumable steps, and per-line rejection.
• Uses a double-entry ledger and an outbox so state, money and events stay consistent.
• Designs reconciliation as a three-way match with exception handling.

Staff (L6+)

• Frames correctness as invariants (balanced ledger, one batch line per capture) and shows how each is enforced and monitored.
• Plans for the operational reality: network deadlines, stuck batches, hot merchants, multi-currency, audit and PCI scope.
• Weighs business trade-offs (lost sale vs duplicate charge, payout timing vs risk) and says who owns each exception queue.