Question: Design review: rolling out a new model version
Companies: Anthropic
Difficulty: Medium
Problem statement
You are reviewing a launch plan written by the serving team. Critique it: rank the risks, propose changes, and say what must be true before you would approve it. You have 45 minutes.
Launch plan: model v2 to all API traffic (draft)
Why. v2 beats v1 on our internal eval suite by 3 points on average. Customers have asked for it.
Plan. On Friday at 10:00 we change the latest alias from v1 to v2 for all customers in all three regions with one config push. Customers who pinned v1 are moved to v2 as well, because v1 will be deprecated.
Capacity. v2 has 1.6 times as many parameters as v1. We keep the same GPU fleet: v2's batching is more efficient, so throughput should be similar. To make room, v1 replicas are scaled to zero right after the switch. The fleet peaks at about 70% utilization today.
Caching. Prompt-cache keys are a hash of the prompt prefix, so cached prefixes keep working after the switch.
Rollback. Push the old config. v1 replicas take about 40 minutes to load weights and warm up.
Monitoring. p50 latency and the 5xx error rate, on the existing dashboard.
Safety. The safety evaluation for v2 is scheduled for the week after launch.
How to run the review
A launch plan is a bet that nothing unexpected happens. The review asks: when something does, how much traffic sees it, how fast do we notice, and how fast can we undo it? Score every part of the plan on those three questions: blast radius, detection, reversal.
- Restate what must stay true: customers' requests succeed at today's latency, answers are at least as good for every major use, safety commitments hold, and pinned customers get what they pinned.
- Check the capacity numbers before anything else. They decide whether this plan fails on its first peak.
- Rank the flaws. Several are launch blockers; say which ones and why, then propose the plan you would approve.
- Credit what is good. A single alias makes the switch and the rollback mechanically simple, and the eval suite exists. Keep both.
The capacity check
v2 has 1.6× the parameters, so it does about 1.6× the compute per token, and its KV cache per token is likely larger too, which means fewer sequences fit in a batch. The fleet peaks at 70% utilization today:
- Same traffic on v2: 70% × 1.6 = 112% at peak. Even if batching really improves throughput by 10%, that is 112 / 1.1 ≈ 102%. Queues grow without bound at peak, latency climbs, and timeouts turn into retries that add more load.
- Scaling v1 to zero frees GPUs, but v2 needs them all at once, so there is no headroom for the rollback either.
- The claim "batching is more efficient" needs a load test at production prompt and output lengths before launch, not after.
To hold 70% peak utilization on v2, the fleet needs about 1.6× the GPUs (or 1.45× if the 10% batching gain is real). If those GPUs don't exist, the launch needs rate limits or a smaller first audience.
Planted flaws
| # | Flaw | Why it matters | Severity | Fix |
|---|---|---|---|---|
| 1 | One config push to 100% of traffic in all regions | Any problem hits every customer at once. There is no comparison group, so even detecting a quality regression is hard | Critical | Staged rollout: internal traffic, then 1%, 5%, 25%, 50%, 100% of latest, one region first, with automatic checks at each stage |
| 2 | v1 scaled to zero right after the switch; rollback takes 40 minutes | At peak, a bad v2 means 40 minutes of degraded service for everyone. "Push the old config" only works if there is something to send traffic to | Critical | Keep v1 at full capacity until v2 has been at 100% for a week; rollback becomes a 1-minute alias flip. Practice the rollback before launch |
| 3 | Capacity assumed, not measured | 112% of peak capacity (above). The first busy hour overloads the fleet | Critical | Load test v2 at peak traffic shape; add GPUs or admission control; launch outside peak |
| 4 | Prompt-cache keys don't include the model version | A cached prefix holds v1's attention state. Reusing it in v2 is not "keeps working"; it is wrong output, silently, for every cache hit | Critical | Include model version (and tokenizer version) in the cache key; accept a cold cache at switch time and size for it |
| 5 | Safety evaluation after launch | Shipping before the safety gate breaks the commitment the gate exists for, and any finding becomes an incident instead of a fix | Critical | Safety eval is a launch blocker; no exceptions in the plan |
| 6 | Pinned v1 customers moved to v2 |
Pinning exists so customers control upgrades. Moving them breaks the contract and their own evals | High | Respect pins; announce the v1 deprecation date with notice, migration guide and a period of overlap |
| 7 | Monitoring is p50 latency and 5xx only | Quality regressions return HTTP 200. p50 hides the tail that times out. Nothing compares v1 and v2 | High | Per-version dashboards: p50/p99 latency, time to first token, tokens per second, error and timeout rates, refusal rate, output length, tool-call and JSON-format failure rates, and complaints. Compare canary with control |
| 8 | "+3 points on average" | An average hides a regression on one category (code, a language, tool use) that matters to specific customers | Medium | Per-category deltas with confidence intervals; a set of must-not-regress evals; replay a sample of real (consented or synthetic) traffic through both versions |
| 9 | Friday 10:00, all regions | The first weekend is when fewest people are around to respond, and a global change has no unaffected region to fail over to | Medium | Launch early in the week, one region first, with an on-call rota for the first 72 hours |
The plan you would approve
| Stage | Traffic on v2 | Hold for | Advance only if |
|---|---|---|---|
| 0 | Internal and shadow traffic (v2 answers, v1 serves) | 2–3 days | Safety and capability gates passed; load test at peak passed; rollback rehearsed |
| 1 | 1% of latest, one region |
24 hours | No significant change in errors, p99 latency, refusals, format failures; no new complaint categories |
| 2 | 5%, then 25% | 24 hours each | Same checks, plus capacity headroom above 20% at peak |
| 3 | 50%, then 100% of latest, region by region |
48 hours | Same checks, holding through a weekday peak |
| 4 | v1 capacity released | after a week at 100% | Pins migrated by customers, deprecation notice period started |
Why 1% is enough to detect a real problem. At 20,000 requests/s, 1% is 200 requests/s, or 720,000 an hour. If the error rate is 0.1%, that is about 720 errors an hour; a doubling to 1,440 is obvious within minutes. Quality signals (refusal rate, format failures) need more volume, which is why stage 1 holds for a day.
Automatic rollback triggers. Error or timeout rate 1.5× the control group, p99 latency 1.3× control, or format failures above a fixed threshold, sustained for 10 minutes: flip the alias back without waiting for a human.
Questions to ask the author
- What did the load test show, at what prompt and output lengths?
- Which customers depend on exact behaviors (JSON mode, tool calls, long context), and are there evals for those?
- How long does the cache take to warm, and what does a cold cache do to latency and cost?
- What is the deprecation policy we have published for pinned versions?
- Who decides to advance a stage, and who can stop it?
Interviewer expectations
Mid-level (L4)
- Objects to the all-at-once switch and proposes a canary with monitoring.
- Notices that rollback takes 40 minutes and that safety evals come after launch.
- Suggests better metrics than p50 and 5xx.
Senior (L5)
- Does the capacity math (112% of peak) and catches the prompt-cache key that silently mixes model versions.
- Proposes a staged rollout with stage gates, warm v1 capacity, respected pins and per-version dashboards with quality signals.
- Ranks the blockers clearly and separates them from the nice-to-haves.
Staff (L6+)
- Turns the critique into an approvable plan: stages, hold times, automatic rollback triggers, and who owns each decision.
- Covers the customer and contract side (pins, deprecation notice, communication) and the eval side (slices, traffic replay, must-not-regress sets).
- Explains how to detect quality regressions statistically at each stage, and plans the GPU budget for running both versions during the rollout.