0

Design review: usage billing for an LLM API

by Frontier Interview· about 4 hours ago· ⑂ 0 remixes

Company: Anthropic
Difficulty: Medium

Problem statement

You are reviewing a teammate's design doc before it goes to the wider team. Read it, then lead a critique: what is wrong, how bad each problem is, what you would change, and what the doc is missing. You have 45 minutes.

Usage billing for the API (draft)

Goal. Bill API customers per token, monthly, and cut off prepaid accounts when their balance runs out. Traffic today: 20,000 requests per second at peak across 50 gateway instances, about 1,500 tokens per request.

Counting. Each gateway keeps an in-memory map from API key to tokens used. Every 60 seconds it flushes the map to Postgres with UPDATE usage SET tokens = tokens + $n WHERE api_key = $k AND month = $m, then clears the map.

Streaming. For streamed responses the gateway counts tokens when the stream completes. If the client disconnects first, nothing is counted.

Retries. Clients retry on timeouts; each retry is a new request and is billed as one.

Months. The month for a request is computed from the gateway's local clock and time zone (gateways run in three regions).

Pricing. At invoice time, a job multiplies each customer's monthly tokens by the current price per token (a float in dollars) and rounds to the cent.

Invoices. The invoice job runs at 00:00 on the 1st, reads usage, and emails invoices.

Prepaid accounts. A daily job at 02:00 subtracts the day's usage from each prepaid balance and disables keys whose balance is below zero.

Monitoring. A dashboard shows total revenue per day.

How to run the review

A design review is graded on prioritization as much as on finding problems. Spend the first five minutes restating the goal and checking the numbers, then go through the doc in order of money at risk, not in the order it is written.

  1. Restate the requirements the doc implies. Bills must be complete (every token billed once), correct (the right price), explainable (a customer can audit an invoice) and timely. Prepaid accounts need a hard limit.
  2. Do the arithmetic. 20,000 requests/s × 1,500 tokens = 30M tokens/s at peak, about 600k tokens/s per gateway. At an illustrative blended $3 per million tokens, that is $90 per second of traffic.
  3. Walk the money path from request to invoice and ask, at each hop: what happens on a crash, a retry, a disconnect, a deploy, a midnight?
  4. Rank and fix. Give each flaw a severity and the smallest change that removes it.
  5. Say what is good. The doc is short, the flush batches writes sensibly, and counting at the gateway is the right place to start.

Planted flaws

# Flaw Why it matters Severity Fix
1 UPDATE ... WHERE month = $m with no insert The first flush of each month, and every new key, matches zero rows. The update "succeeds" and the usage is gone. Every customer's first minute of each month, and all usage of any new key, is never billed. Critical An upsert (INSERT ... ON CONFLICT DO UPDATE), or better, the event log in the revised design
2 Usage lives in gateway memory for up to 60 s A crash or deploy drops up to a minute of that gateway's usage: 600k tokens/s × 60 s = 36M tokens, about $108 per crash. A rolling deploy that doesn't drain loses about 30 s on each of 50 gateways: around $2,700 per deploy. A failed flush also clears the map. Critical Emit one durable usage event per request (request id, key, model, input and output tokens, UTC time, price version) to a replicated log before acknowledging the response
3 Prepaid balances are checked once a day A leaked key at 1,000 requests/s spends 1.5M tokens/s, $4.50 per second, about $390,000 before the 02:00 job notices Critical A near-real-time spend counter checked on the request path: reserve an estimate before the request, settle after; hard per-key and per-account spend caps
4 Disconnected streams are not billed The GPUs already generated the tokens. A client that disconnects just before the end gets everything for free, and normal network drops are lost revenue High Count tokens as they are generated; bill what was produced when the stream ends for any reason
5 Price applied at invoice time from the current table A price change on the 20th re-prices the first 19 days. Different models have different prices, but usage only stores a token count High Store model and price version on each event; price at event time; separate input and output tokens
6 No reconciliation Nothing compares billed tokens with what the model servers actually generated, so flaws 1, 2 and 4 would never be noticed. A revenue dashboard only shows drops after the fact High A daily job comparing gateway events with model-server counts per key and hour; alert on a gap above 0.1%
7 Retries billed as new requests A client that times out after the server finished pays twice for one answer Medium Idempotency keys: a retry with the same key returns the stored result instead of generating and billing again
8 Month taken from each gateway's local time zone Requests around midnight land in different months depending on the region that served them Medium UTC everywhere; apply the customer's billing time zone once, when aggregating
9 Float dollars, rounded per invoice Floats make totals depend on summation order, and invoices are not reproducible to the cent Medium Integer micro-dollars (or Decimal), one documented rounding step per invoice line
10 Invoices emailed at 00:00 on the 1st Late events (flush lag, retries, clock skew) miss the invoice; a rerun of the job emails customers twice Medium Close the month after a grace period (for example 6 hours), make invoice generation idempotent per account and month, review before sending

Flaws 1 and 2 together are the headline: this design under-bills every month, and nothing in it would tell you.

Revised design

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"gw","x":0,"y":0,"width":180,"height":70,"label":{"text":"API gateway"}},
 {"type":"ellipse","id":"log","x":260,"y":-5,"width":200,"height":80,"label":{"text":"Usage event\nlog"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"agg","x":540,"y":0,"width":200,"height":70,"label":{"text":"Aggregator\ndedupe by id"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"ellipse","id":"store","x":820,"y":-5,"width":200,"height":80,"label":{"text":"Hourly usage"}},
 {"type":"rectangle","id":"inv","x":1100,"y":0,"width":180,"height":70,"label":{"text":"Invoicer"}},
 {"type":"rectangle","id":"limit","x":0,"y":170,"width":180,"height":70,"label":{"text":"Spend limiter"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"recon","x":820,"y":170,"width":200,"height":70,"label":{"text":"Reconciler"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":80,"height":0,"start":{"id":"gw"},"end":{"id":"log"}},
 {"type":"arrow","id":"a2","x":460,"y":35,"width":80,"height":0,"start":{"id":"log"},"end":{"id":"agg"}},
 {"type":"arrow","id":"a3","x":740,"y":35,"width":80,"height":0,"start":{"id":"agg"},"end":{"id":"store"}},
 {"type":"arrow","id":"a4","x":1020,"y":35,"width":80,"height":0,"start":{"id":"store"},"end":{"id":"inv"}},
 {"type":"arrow","id":"a5","x":90,"y":70,"width":1,"height":100,"start":{"id":"gw"},"end":{"id":"limit"}},
 {"type":"arrow","id":"a6","x":920,"y":75,"width":1,"height":95,"start":{"id":"store"},"end":{"id":"recon"}},
 {"type":"text","id":"legend","x":0,"y":290,"text":"Shaded: what the review adds. Every request becomes one durable, priced event.","fontSize":16}
]}
  • Gateway writes one usage event per request to a replicated log (Kafka or similar) and only then finishes the response. Event: request_id, account, key, model, input_tokens, output_tokens, price_version, finished_at (UTC), status. At 20,000 requests/s and about 200 bytes per event, that is 4 MB/s, about 350 GB a day: modest for a log, and it is the audit trail customers ask for when they dispute an invoice.
  • Aggregator consumes the log, drops duplicate request_ids, prices each event with its price version, and writes per-key hourly rollups in integer micro-dollars. Rerunning it over the log rebuilds the same totals.
  • Spend limiter sits on the request path: a per-account counter (in memory per region, synced every second) with a reservation before generation and settlement after. Prepaid accounts stop within seconds of reaching zero, not the next day.
  • Reconciler compares rollups with model-server token counts every hour and alerts on drift.
  • Invoicer closes the month after a grace period, generates idempotently, and sends after review.

Questions to ask the author

  • What are the published prices: per model, input versus output, any tiers or discounts? Who changes them, and how often?
  • What do customers see today: a usage API, a dashboard? Invoices must match it to the cent.
  • How do refunds, credits and failed requests (a 500 after generating) get billed?
  • What is the acceptable gap between usage and enforcement for prepaid accounts: one second, one minute?
  • Who is paged when billing breaks, and how would they know?

Interviewer expectations

Mid-level (L4)

  • Finds the obvious losses: in-memory counts on a crash, unbilled disconnects, float money.
  • Proposes durable per-request records and integer money.
  • Communicates the critique clearly and constructively.

Senior (L5)

  • Finds the missing insert, the daily prepaid check and the price-at-invoice-time problem, and ranks them with dollar estimates.
  • Proposes the event log with idempotent aggregation, price versions and a spend limiter on the request path.
  • Asks about reconciliation and late events, and fixes the month boundary.

Staff (L6+)

  • Leads with what matters most (silent under-billing and unbounded prepaid spend), quantifies each, and keeps minor issues brief.
  • Designs reconciliation against an independent source and the alerting around it, and covers disputes, audit and customer-facing usage.
  • Offers a migration path from the current design (dual-write events, compare for a month, switch invoicing) and says what to ship first.

Sign in to comment, vote or remix.