A new model lands, the benchmark table goes up, and a hundred posts declare a fresh state of the art. None of it tells you whether the thing can do your job. Reading a model card like an operator means translating the vendor’s framing into your own and discounting everything that does not map to a task you actually run.

The discipline is not new. The paper that introduced model cards in 2019 already asked for exactly this: performance reported across a range of conditions, plus a clear statement of intended use, never a single headline score. What has changed is that the headline score has gotten less trustworthy, not more, so the old advice is now urgent.

Separate the claim from the conditions

Every headline number was produced under conditions: a prompt format, a temperature, a number of attempts, sometimes a generous rubric. The number is real; the conditions are doing quiet work. The clearest recent example is competitive coding. On LiveCodeBench Pro, a benchmark built and graded by olympiad medalists, the best model scored 53 percent on medium problems and zero percent on the hard ones, and the authors traced the strong aggregate scores to implementation precision and tool use rather than to better reasoning. The models that looked brilliant on the summary line were, on the hard cases, confidently wrong. So before you trust a score, ask how many attempts it took and whether the harness resembles anything you would ever run. A vendor reporting pass@k, the best of several tries, is not lying. It is reporting a different job than the single shot you will run in production.

Map each benchmark to a job, not to other models

A leaderboard ranks models against models. You need models against tasks. This is the part the original model-card authors got right and most launch posts skip: a useful report measures performance under the specific conditions a deployer cares about, not one tidy average. Stanford’s HELM project made the same argument at scale, scoring models on seven separate dimensions, including accuracy, calibration, robustness, and fairness, across dozens of scenarios, after finding that earlier evaluations had each covered, on average, under a fifth of the scenarios that mattered. The operator version is humbler: write down the three jobs you would actually hand a model, then ask which reported benchmark, if any, resembles each. Often the answer is none, and that absence is itself the finding.

Distrust the answer key, and the clock

Two quieter problems sit underneath the numbers. The first is contamination. When researchers built a fresh clone of the GSM8k grade-school math test, called GSM1k, several model families dropped by as much as 13 percent, and the size of the drop tracked how likely a model was to reproduce the original questions. Some families had partly memorized the test. Frontier models mostly had not, which is the honest nuance. The second problem is the answer key itself. A careful re-annotation of MMLU, the field’s default knowledge benchmark, estimated that 6.49 percent of its questions contain errors, and in the virology subset more than half were flawed: wrong ground truth, several correct answers, or no answerable question at all. Fixing them shifted the reported rankings. And whatever the number is, it spoils fast: the AI Index reports that on benchmarks designed to be hard, scores leapt 18.8, 48.9, and 67.3 points in a single year. A record-setting result is a perishable good.

Treat the model card as a hypothesis, not a verdict. The vendor proposes; your own eval set decides.

Price the failure mode, not the average

Average accuracy hides the shape of the errors, and the shape is what costs you. A model wrong five percent of the time in obvious, catchable ways is far safer than one wrong three percent of the time in fluent, plausible ways, because the second kind slips past review. This is the operator’s real metric: not raw accuracy but the recoverable error rate, how often a mistake is one you will catch before it reaches a customer or a ledger. LiveCodeBench Pro named the trap precisely. The dangerous failures were the confident ones.

Test on your own data before you believe anyone

The only benchmark that settles the question is fifty examples from your own work, graded by someone who knows the right answer. It takes an afternoon, and it beats every leaderboard, because it measures the one distribution you care about: yours. Everything above is just the reason you cannot skip it. The published numbers are real, but they were never about your task. Refusing to outsource “is this good enough for me” to a number that answered a different question is the entire discipline.