Design review: usage billing for an LLM API

You are reviewing a teammate's design doc before it goes to the wider team. Read it, then lead a critique: what is wrong, how bad each problem is, what you would change, and what the doc is missing. You have 45 minutes.

Usage billing for the API (draft)

Goal. Bill API customers per token, monthly, and cut off prepaid accounts when their balance runs out. Traffic today: 20,000 requests per second at peak across 50 gateway instances, about 1,500 tokens per request.

Counting. Each gateway keeps an in-memory map from API key to tokens used. Every 60 seconds it flushes the map to Postgres with UPDATE usage SET tokens = tokens + $n WHERE api_key = $k AND month = $m, then clears the map.

Streaming. For streamed responses the gateway counts tokens when the stream completes. If the client disconnects first, nothing is counted.

Retries. Clients retry on timeouts; each retry is a new request and is billed as one.

Months. The month for a request is computed from the gateway's local clock and time zone (gateways run in three regions).

Pricing. At invoice time, a job multiplies each customer's monthly tokens by the current price per token (a float in dollars) and rounds to the cent.

Invoices. The invoice job runs at 00:00 on the 1st, reads usage, and emails invoices.

Prepaid accounts. A daily job at 02:00 subtracts the day's usage from each prepaid balance and disables keys whose balance is below zero.

Monitoring. A dashboard shows total revenue per day.