Question: Design review: an eval gate for model launches
Companies: Anthropic
Difficulty: Medium
Problem statement
You are reviewing the design of the evaluation gate that decides whether a new model checkpoint can launch. Critique it: what it can miss, how it can mislead, what you would change, and what is missing. You have 45 minutes.
Launch eval gate (draft)
Gate. Run the candidate on our 12 benchmarks. It may launch if its average score across the 12 is at least the current model's average.
Sampling. Each benchmark item is answered once, at temperature 1.0, to match production settings.
Grading. Multiple-choice items are graded by exact match. Free-form answers are graded by the candidate model itself, since it is our strongest model.
Data. The benchmarks are public sets (300–1,000 items each). Pretraining data is a web crawl up to last month.
Harness. Each benchmark's prompt template is written by whoever adds the benchmark. Harness changes merge to main like any other code. Results go into a shared spreadsheet.
Cost. A full run takes 9 hours on the eval cluster, so we run it once, on the final checkpoint.
How to run the review
A launch gate has two jobs: block a model that is worse in a way that matters, and not block a model that is better. Review the doc against both, and against a third question that matters as much: can the result be trusted next month, when someone compares against it?
- Name the decisions the gate makes and their costs. Shipping a regression on safety or on a major customer use case is far worse than delaying a good model by a few days.
- Check the statistics. Most of the flaws here are about noise: how big a difference the gate can actually see.
- Check the inputs: contamination, grading, prompts, versions.
- Rank, fix, and propose the gate you would sign off on.
- Credit what is good: a fixed suite, run automatically, with a written rule. Many teams launch on vibes.
The noise in the numbers
A benchmark with 500 items at 70% accuracy has a standard error of √(0.7 × 0.3 / 500) ≈ 2.0 points, so its 95% interval is about ±4 points. Two models each measured that way, compared unpaired, have a difference interval of about ±5.7 points. The doc's "+3 points" on one benchmark would be noise.
Pairing helps a lot, because both models answer the same items: only the items where they disagree carry information. The code below computes the paired interval for a typical case: on 500 items, the new model fixes 30 that the old one missed and breaks 20 it got right.
import math
def paired_diff_ci(old, new, z=1.96):
"""95% interval for accuracy(new) - accuracy(old) graded on the same
items; old and new are lists of 0/1 per item."""
diffs = [b - a for a, b in zip(old, new)]
n = len(diffs)
mean = sum(diffs) / n
var = sum((d - mean) ** 2 for d in diffs) / (n - 1)
half = z * math.sqrt(var / n)
return mean - half, mean + half
# 500 items: the new model fixes 30 the old one missed and breaks 20 it got right.
old = [0] * 30 + [1] * 20 + [1] * 330 + [0] * 120
new = [1] * 30 + [0] * 20 + [1] * 330 + [0] * 120
lo, hi = paired_diff_ci(old, new)
print(f"old {sum(old) / 500:.3f}, new {sum(new) / 500:.3f}, difference 95% CI [{lo:+.3f}, {hi:+.3f}]")
It prints old 0.700, new 0.720, difference 95% CI [-0.008, +0.048]: a 2-point gain that is still not distinguishable from zero. Sampling once at temperature 1.0 adds more noise on top, since each item's result is itself a coin flip.
Planted flaws
| # | Flaw | Why it matters | Severity | Fix |
|---|---|---|---|---|
| 1 | Gate on the average across 12 benchmarks | Benchmarks are not interchangeable. +6 points on 11 of them and −30 on the safety set averages to +3 and passes. The gate can approve exactly the regression it exists to catch | Critical | A per-benchmark rule: must-pass sets (safety, key customer tasks) with hard thresholds, and non-regression on each benchmark judged by its interval, not its point estimate |
| 2 | The candidate grades its own free-form answers | Models favor their own style and phrasing, and a model that grades leniently passes itself. The grader changes with every candidate, so scores are not comparable across launches | Critical | A fixed, versioned grader (programmatic where possible; otherwise a pinned judge model calibrated against human labels on a sample, with agreement reported) |
| 3 | One sample per item at temperature 1.0, compared by point estimate | Noise of ±4 to ±6 points per benchmark (above). The gate's decisions are partly random | High | Paired comparisons with confidence intervals; several samples per item (or greedy decoding for capability, sampled runs for behavior); grow small sets |
| 4 | Public benchmarks, web crawl up to last month | Test items leak into pretraining data. Scores go up because the model has seen the answers, and the gate rewards it | High | Decontaminate training data (n-gram overlap against every eval set), keep private held-out sets, add canary strings, and report scores on fresh items separately |
| 5 | Prompt templates written ad hoc; harness unversioned; results in a spreadsheet | A template change can move a score by several points. With no versions, last quarter's number was measured differently from today's, so the comparison is meaningless | High | Version the harness, templates, grader and datasets together; store every result with that version and the model id; re-run the current model on the same harness version for every comparison |
| 6 | Run only on the final checkpoint | A regression is found at the end, when it is most expensive to fix, and there is no trend to show when it started | Medium | A fast subset (about 30 minutes) on every checkpoint; the full 9-hour suite on release candidates |
| 7 | Only benchmarks | No human evaluation, no latency or cost, no checks of behaviors customers depend on (format following, tool use, refusals) | Medium | Add human preference evals on real-task prompts, product-behavior evals, and serving cost and latency to the gate |
The gate you would approve
| Check | Rule | Owner |
|---|---|---|
| Safety and policy evals | Must pass fixed thresholds; any failure blocks | Safety team |
| Must-not-regress task sets (code, tool use, long context, top customer workflows) | Paired 95% interval of the difference must not be entirely below −1 point | Each area's owner |
| Broad benchmarks | Reported with intervals; a significant regression needs a written sign-off | Eval team |
| Human preference on real-task prompts | Win rate against the current model with interval | Eval team |
| Contamination report | Overlap rate per eval set below threshold; fresh-item scores reported | Data team |
| Reproducibility | Every number tied to harness, template, grader and dataset versions | Eval team |
The final decision is a human sign-off on this report, not an automatic average.
Questions to ask the author
- Which of the 12 benchmarks would block a launch on their own if they regressed? If the answer is "some of them", the average can't be the rule.
- How was the current model's score measured: same harness version, same templates, same grader?
- What is the item count per benchmark, and what difference are we trying to detect?
- How do we know the free-form grader agrees with humans?
- What happened at the last launch: did anything regress that this gate would have missed?
Interviewer expectations
Mid-level (L4)
- Objects to self-grading and to averaging away a regression on an important benchmark.
- Mentions contamination of public benchmarks.
- Suggests running evals more often than once at the end.
Senior (L5)
- Quantifies noise (about ±4 points per 500-item benchmark) and proposes paired comparisons with intervals and more samples.
- Proposes must-pass sets with per-benchmark rules, a fixed calibrated grader, decontamination and versioning.
- Ranks the flaws and keeps the critique constructive.
Staff (L6+)
- Designs the full gate: what blocks, who owns each check, how close calls are decided, and how results stay comparable across launches.
- Connects the gate to production evidence (staged rollout, traffic replay) and to how evals themselves are maintained (fresh items, rotation, contamination tracking).
- Explains the asymmetric costs of false passes and false blocks and sets thresholds from them.