Method

Where numbers come from#

Every result traces to one of three kinds of document. Bar colour on the charts is the lab; a pale bar with an outline is a number the model's own lab reported.

• Official leaderboard. The benchmark's owner ran the model and published the score on their own board or in their own paper.
• Independent run. A party that neither built the model nor owns the benchmark ran it and published the result with its configuration.
• Vendor-reported. The lab that built the model printed the score in its system card, model card or launch post. The lab chose the settings, and sometimes the grader.

Right now the 436 headline results split as 332 official leaderboard, 45 independent run and 59 vendor-reported. Each board chart can be filtered by who reported the number.

How a row is checked#

A row is published only when all four of these are recorded:

• the document, preferring the first-party original over any page that repeats it;
• the number exactly as the document prints it, with its own rounding and its own percent sign or lack of one;
• a verbatim quote from the document that contains the number and shows which model and which measure it belongs to;
• a locator: page and table, section, figure, or the board's row and the date the board showed it.

The configuration the source states goes into the row as well: reasoning effort, harness, grader, whether a length adjustment was applied. It appears under the model name on every chart and in full in the row's sources. Two rows for the same model on one board almost always differ in configuration, and both are kept.

Printed form and chart form can differ. A card that prints 66.0 on a board scored 0 to 1 is charted at 0.660; the row's sources show both. Nothing is rounded beyond what the source printed.

When documents disagree#

A first-party document outranks a page that mirrors it, and a benchmark owner's board outranks a secondary write-up of that board. The row keeps the number from its document of record.

Other documents that print the same result are listed in the row's sources, each with the number it printed: “also printed in” when it matches, “a different number in” when it does not. When the disagreement cannot be settled from the documents, the row carries the disputed marker.

Reading a board#

Rank means position among the rows on that board and that measure only. Some boards list many more models than this site carries; where the board's full field size is known, pages say so as “N of M models on the board”. A rank here can be higher than the model's rank on the official board for that reason.

Vendor-reported rows on the same board may use different graders or settings from each other. The configuration line under each row is there so a reader can decide whether two rows are comparable for their purpose.

The index#

The Clinical Benchmarks Index is this site's own calculation, made from the board results and nothing else. It gives one ordering for readers who want one; it is not a benchmark, and the boards remain the evidence.

1. Per board. A model's best headline result counts on the board's own 0 to 100 scale: a board scored 0 to 1 is read times 100, a percentage as printed. A board's leader keeps its actual score instead of being stretched to 100, and a new model below the leader changes no one else's figure.
2. Board weight. A board whose leader is close to 100 has little room left to separate models. So boards near their ceiling count for less: weight is the board's headroom (100 minus its leader's score) over 40, between 0.25 and 1. Near their ceiling now: MedScribe (Vals AI): leader 91.4, weight 0.25; MedXpertQA (MM): leader 81.5, weight 0.463; MedPIC: leader 80.7, weight 0.482.
3. Across boards. The weighted average of the model's board scores: each score counts by its board's weight, over the boards the model has results on.
4. Confidence multiplier. A score from fewer boards rests on less evidence, so it is scaled down: full from 6 boards; below that the square root of boards / 6 (5 boards x0.913, 4 boards x0.816, 3 boards x0.707, 2 boards x0.577, 1 board x0.408). It is about how many boards a score rests on, and is separate from the confidence markers on single rows.
5. Score. The weighted average times the multiplier, to one decimal. The working printed with every score multiplies out.

• Who is ranked. Every model with a result on at least one index board. Left out: harness plus model pairings, products, research systems, human and baseline references.
• Which boards. A board counts when it has results from at least 5 such models. The table below gives the reason for each board that does not count.
• Coverage. The pips beside each entry mark the index boards its score rests on, and the index views can be filtered to models with results on 2, 3 or 5 or more boards. A filter hides entries; it changes no score or rank.

Worked example

Every index score is shown with its working. Claude Fable 5 has results on 5 of 10 index boards. The weighted average of its board scores is 66.9. Five boards is fewer than six, so the multiplier is 0.913 and the index is 66.9 × 0.913 = 61.1, rank 3. MedScribe (Vals AI) counts ×0.25 because its leader already scores 91.4.

The method in use took effect on 6 October 2026. The version before it placed each result between its board's lowest and highest result, so every board leader scored 100, and took a plain mean over the boards.

149 models are ranked: 55 from one board, 5 from six or more with no multiplier.

Confidence markers#

Rows that have been matched against their document carry no marker. The other two states are always visible, on charts, tables and model pages:

• Unverified the number has been recorded but not yet matched against its source document.
• Disputed documents disagree about the number; the row's sources show each one.

What we never do#

• Combine, average or rank scores from different benchmarks anywhere except the index, which is labelled as this site's own calculation wherever it appears. Each board ranks only its own rows.
• Add precision a source did not print, or convert a score to another scale and present it as the source's.
• Publish a number without its document. A row whose document is not yet recorded shows “source pending”.

What gets a board#

A benchmark is tracked when it tests clinical work (reasoning, documentation, coding, EHR tasks, administrative work or patient safety) and has recent results for current models. Benchmarks that saturated or stopped reporting current models are listed with the reason at the foot of the boards page. Benchmarks about AI that people use for their own health are covered by the sister site linked in the footer.