Model Leaderboards: The Four Dimensions a Model Is Scored On

What this slice covers

Model leaderboards in the Foundry portal exist to let you compare catalogue models against industry-standard benchmarks instead of vendor marketing. This note covers what the leaderboards measure, how the four scoring dimensions relate to one another, and how to read a comparison without drawing the wrong conclusion. A companion note covers how the numbers are produced and where they break down; here the goal is to understand the scoreboard itself.

Reach the leaderboards from the model catalogue — the view leaderboard and compare options — and reach a single model's figures from the benchmarks tab of its model card.

What is on the board, and what is not

The leaderboards carry a curated selection of text-based language models, not the whole catalogue. Two criteria govern inclusion. Models sold by Azure are prioritised for their relevance to common generative AI scenarios. And a model must handle general-purpose language tasks — reasoning, knowledge, question answering, mathematics, coding — to be comparable at all. Specialised models such as protein-folding systems, and other modalities, are out of scope.

Large and small language models are covered, and embedding models are scored separately against their own quality benchmarks for search and retrieval work.

The practical implication: absence from a leaderboard is not evidence that a model is weak. It may simply be specialised.

The four dimensions

Quality. A quality index averages accuracy scores across a set of standard benchmark datasets covering reasoning, chemistry, scientific reasoning, question answering, coding, general knowledge, and agentic tool-call selection. Values run from zero to one, higher being better. Accuracy is usually computed by exact match — the generated text either matches the reference answer or it does not — with coding datasets instead using a pass rate against unit tests.

Safety. Safety is proxied through the HarmBench suite, reported as an attack success rate across standard harmful behaviours, contextually harmful behaviours, and copyright violations. The direction of the metric inverts here: lower attack success rates indicate more robust models. Two further datasets sit alongside it — one measuring knowledge in sensitive domains such as biosecurity and cybersecurity, where a higher score means more dangerous capability, and one measuring toxic-content detection by F1 score, where higher is better.

Performance. Latency and throughput, reported as means and as percentiles, plus time to first token and time between tokens. These tell you how a model will feel to a user, which the quality index cannot.

Cost. Actual measured cost of running each model across the quality datasets, using the real count of input, reasoning, and output tokens consumed and the reasoning effort used in evaluation. Reported in dollars per benchmark run, lower being better.

Scenario leaderboards

Alongside the aggregate view, benchmark datasets are grouped into scenarios that match real evaluation goals — reasoning, coding, general knowledge, question answering, mathematics, groundedness, toxicity detection, and the safety scenarios. If your workload maps cleanly to one of these, the scenario board is the better starting point than the overall quality index, because a strong all-rounder can still be middling at the one thing you need.

Reading the board like an engineer

The four dimensions are meant to be read together, and the tensions between them are the point:

• A model with a high quality index and high latency may be unsuitable for anything interactive. Performance metrics exist precisely to surface that.
• Safety scores deserve weight proportional to exposure. For a customer-facing application where harmful output is a serious risk, a safety difference can outrank a quality difference.
• Cost is measured on the benchmark workload's token mix, which is unlikely to match yours. Treat it as relative, not as a forecast.

Leaderboards are refreshed as new models and datasets appear, so a comparison you ran last quarter is a historical artefact rather than current advice.

Mistakes to avoid

• Picking the top row of the quality board and stopping. That ignores latency, safety, and price entirely.
• Reading safety scores in the same direction as quality scores. For attack success rate, low is good, and for sensitive-domain knowledge a high score is the bad outcome.
• Comparing an embedding model to a chat model. They are benchmarked on separate scales for separate jobs.
• Concluding that a model missing from the board is inferior rather than out of scope.

What to carry forward

Model benchmarks assess models on quality, safety, cost, and throughput, with embedding models scored on their own quality scale. Use the aggregate view to shortlist, the scenario boards to match your task, and always read at least two dimensions before committing to a model.