How Benchmark Numbers Are Made, and Where They Mislead

What this slice covers

A leaderboard reduces a model to a number, and a number invites trust it has not earned. This note is about the machinery behind those numbers: how Foundry actually computes a benchmark score, what assumptions are baked into each family of measurement, and why the documentation itself devotes whole sections to the limits of its own results. Knowing the dimensions on the board is one skill; knowing what a score cannot tell you is the skill that stops you making a bad model choice.

The evaluation pipeline

Benchmark results come from public datasets that the research community already uses for language-model evaluation, typically hosted in repositories maintained by the people who created or curated them. Foundry's evaluation pipelines download the data from those original sources, extract a prompt from each example row, generate a model response, and compute the relevant accuracy metric.

Three things follow from that description.

First, the questions are public. Anyone — including model builders — can see them.

Second, prompts are constructed following the methodology of each dataset's original paper, and most prompts include several shots: worked examples of complete questions and answers placed before the real question to prime the model. The number of shots varies by dataset, and the examples are drawn from a portion of the data held out from evaluation.

Third, the score depends on that construction. Change the prompt wording or the number of few-shot examples and the number moves. A benchmark measures a model under a particular prompting regime, not in the abstract.

Where quality scores go stale

Benchmark datasets saturate. As models are trained or tuned on data resembling the benchmark, scores climb without the underlying capability improving proportionally, and the gap between the top entries compresses until it stops being informative. This is why new datasets keep being added and why a two-point difference near the top of a mature benchmark should not decide anything.

The metrics themselves are also blunt in ways worth knowing. Exact match rewards producing the reference string; a correct answer phrased differently scores zero. Coding datasets use a pass rate against unit tests, which is a far better proxy for the real task but only measures the tests that exist.

Where performance and cost numbers go stale

Performance figures are collected under a fixed, artificial regime: synthetic prompts built from static text, a set input-to-output token ratio, requests sent sequentially rather than concurrently, single-region deployments, specified regions and rate limits, aggregated over a fortnight of scheduled trials. That regularity is what makes models comparable to each other. It is also exactly what your production traffic is not.

Real latency varies with workload pattern, concurrency, region, and deployment configuration. Read published latency as a ranking signal between models, never as a service-level prediction for your application.

Cost carries a comparable caveat. Where an estimated figure uses a fixed input-to-output ratio, that ratio is an assumption about a workload rather than a measurement of yours; and any cost number reflects pricing at the time it was captured.

Where safety scores are weakest

The documentation is unusually candid here, and the candour is examinable. Safety is multidimensional, and the limitation is stated plainly: no single open-source benchmark can capture a system's full safety across all scenarios. Many benchmarks saturate, or are misaligned between how the benchmark was designed and how the risk is actually defined. Some lack clear documentation of how the target risk was conceptualised at all. The consequence is that safety results can both overestimate and underestimate real-world behaviour.

There is a methodological detail that matters too: safety benchmarking is largely performed with Foundry Guardrails — previously called content filters — turned off, so the scores describe the bare model, not the system you would actually deploy behind guardrails.

What to do instead of trusting the board

Leaderboards are a shortlisting tool, not a decision tool. The intended workflow is to use them to narrow the field, then evaluate the shortlist on your own data and your own task. Foundry's evaluation tooling for generative AI applications exists for precisely that step, and the benchmark documentation points you toward it explicitly.

Practically: define what a good answer looks like for your use case, assemble a modest set of representative inputs, run your two or three candidates against it, and measure. Twenty real examples from your domain will tell you more than any public index.

Mistakes to avoid

• Treating a benchmark score as a property of the model rather than of a model plus a prompting regime.
• Ranking models on differences smaller than the noise a saturated dataset produces.
• Quoting published latency in a design document as though it were a guarantee.
• Assuming safety scores describe a deployed system when guardrails were disabled during measurement.

What to carry forward

Every benchmark encodes assumptions — about prompts, about workloads, about what the risk even is. Learn those assumptions, use the boards to shortlist, and let your own evaluation make the final call.