Design review: an eval gate for model launches

You are reviewing the design of the evaluation gate that decides whether a new model checkpoint can launch. Critique it: what it can miss, how it can mislead, what you would change, and what is missing. You have 45 minutes.

Launch eval gate (draft)

Gate. Run the candidate on our 12 benchmarks. It may launch if its average score across the 12 is at least the current model's average.

Sampling. Each benchmark item is answered once, at temperature 1.0, to match production settings.

Grading. Multiple-choice items are graded by exact match. Free-form answers are graded by the candidate model itself, since it is our strongest model.

Data. The benchmarks are public sets (300–1,000 items each). Pretraining data is a web crawl up to last month.

Harness. Each benchmark's prompt template is written by whoever adds the benchmark. Harness changes merge to main like any other code. Results go into a shared spreadsheet.

Cost. A full run takes 9 hours on the eval cluster, so we run it once, on the final checkpoint.