Clinical Benchmarks
Menu

PhysicianBench

Agents carrying out long-horizon physician workflows inside real EHR systems, verified by execution against those systems.

academic team28 rows20 models on the boardmeasured to Oct 2026

Official pagePaper
More about this boardLess

Agents carrying out long-horizon physician workflows inside real EHR systems, verified by execution against those systems.

LLM agents on long-horizon composite physician workflows inside real EHR environments, with execution-grounded verification against actual EHR systems via standard commercial APIs.

Scores come from the paper; there is no standalone public leaderboard, and the results also feed the ARISE MAST composite. The lower half of the table falls steeply, with several agents near 1 percent on Pass^3 consistency.

Published by Academic team (Ruoqi Liu, Imran Q. Mohiuddin et al., arXiv 2605.02240); also a MAST component, released May 2026. 100 real-world clinical tasks, 21 specialties, 670 structured checkpoints (~27 tool calls per task). Pass@1 success rate %, higher better (3 independent runs; Pass^3 also reported).

Table
Table 2, p. 8
Headline metric
Pass@1 (%), mean ± sd over 3 runs
Paper date
4 May 2026
Paper version
v1
Models in the paper's table
12

Ranking

  1. 1Claude Opus 5.5 (max)Anthropic · Anthropic harness68.4
    Printed as 68.4%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Opus 5.5 (max).
    PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
    Every result from this document

    Link to this row

  2. 2Claude Sonnet 5.5 (max)Anthropic · Anthropic harness63.2
    Printed as 63.2%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Sonnet 5.5 (max).
    PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
    Every result from this document

    Link to this row

  3. 3Claude Fable 5.1 (max)Anthropic · Anthropic harness61.0
    Printed as 61.0%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Fable 5.1 (max).
    PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
    Every result from this document

    Link to this row

  4. 4Claude Opus 5 (max)Anthropic · Anthropic harness57.6
    Printed as 57.6%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Opus 5 (max).
    PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
    Every result from this document

    Link to this row

  5. 5Claude Sonnet 5.5 (xhigh)Anthropic · Anthropic harness56.4
    Printed as 56.4%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; xhigh effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. p. 138, Figure 8.15.B, right panel PhysicianBench (pass@1); orange Sonnet 5.5 xhigh label (visually read printed labels).
    Sonnet 5.5 | PhysicianBench (pass@1) | xhigh 56.4%
    Every result from this document

    Link to this row

  6. 6Claude Sonnet 5.5 (high)Anthropic · Anthropic harness47.6
    Printed as 47.6%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; high effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. p. 138, Figure 8.15.B, right panel PhysicianBench (pass@1); orange Sonnet 5.5 high label (visually read printed labels).
    Sonnet 5.5 | PhysicianBench (pass@1) | high 47.6%
    Every result from this document

    Link to this row

  7. 7GPT-5.5OpenAI · pass@146.3
    Printed as 46.3 ± 1.2Official leaderboard, measured May 2026Configuration: pass@1; Pass^3 28.0
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
    GPT-5.5 46.3 ± 1.2 57.4 28.0 41.9
    Every result from this document

    Link to this row

  8. 8Claude Haiku 5.5 (max)Anthropic · Anthropic harness43.0
    Printed as 43.0%Vendor-reported, measured Oct 2026Configuration: Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 1 and 2
    Claude Haiku 5.5 System Card system card, Anthropic, 7 Oct 2026. pp. 132-133, Section 8.11.2, Figure 8.11.2.A caption and paragraph; also p. 133, Section 8.11.3
    PhysicianBench is a public benchmark of 100 physician tasks carried out in an electronic health record (EHR). Claude Haiku 5.5 passed 43.0% of attempts (five attempts on each task).
    Every result from this document

    Link to this row

  9. 9Claude Sonnet 5 (max)Anthropic · Anthropic harness37.4
    Printed as 37.4%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Sonnet 5 (max).
    PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
    Every result from this document

    Link to this row

  10. 10Claude Haiku 5.5 (xhigh)Anthropic · Anthropic harness35.8
    Printed as 35.8%Vendor-reported, measured Oct 2026Configuration: Anthropic harness; 100 tasks; pass@1; xhigh effort; Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 5
    Claude Haiku 5.5 System Card system card, Anthropic, 7 Oct 2026. p. 133, Section 8.11.3; p. 134, Figure 8.11.3.A PhysicianBench panel
    On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
    Every result from this document

    Link to this row

All 28 rowsTop 10 only
  1. 11GPT-6 Luna (Anthropic run)OpenAI · Run by Anthropic through…33.6
    Printed as 33.6%Independent run, measured Oct 2026Configuration: Run by Anthropic through OpenAI's public API; Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; not OpenAI's published figure
    Claude Haiku 5.5 System Card system card, Anthropic, 7 Oct 2026. p. 133, Section 8.11.2; p. 132, Figure 8.11.2.A GPT-6 Luna bar; p. 131-132, Section 8.11.1 says GPT-6 Luna scores come from Anthropic's own runs
    This was 9.4 points above GPT-6 Luna at 33.6% (95% confidence interval 2.6 to 16.2 points, paired by task)
    Every result from this document

    Link to this row

  2. 12Claude Opus 4.6Anthropic · Pass^3 18.031.7
    Printed as 31.7 ± 2.3Official leaderboard, measured May 2026Configuration: Pass^3 18.0
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
    Claude Opus 4.6 31.7 ± 2.3 41.5 18.0 25.2
    Every result from this document

    Link to this row

  3. 13Claude Haiku 5.5 (high)Anthropic · Anthropic harness31.6
    Printed as 31.6%Vendor-reported, measured Oct 2026Configuration: Anthropic harness; 100 tasks; pass@1; high effort; Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 5
    Claude Haiku 5.5 System Card system card, Anthropic, 7 Oct 2026. p. 133, Section 8.11.3; p. 134, Figure 8.11.3.A PhysicianBench panel
    On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
    Every result from this document

    Link to this row

  4. 14Claude Sonnet 5.5 (medium)Anthropic · Anthropic harness30.0
    Printed as 30.0%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; medium effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. p. 138, Figure 8.15.B, right panel PhysicianBench (pass@1); orange Sonnet 5.5 medium label (visually read printed labels).
    Sonnet 5.5 | PhysicianBench (pass@1) | medium 30.0%
    Every result from this document

    Link to this row

  5. 15Claude Opus 4.7Anthropic · Pass^3 18.029.3
    Printed as 29.3 ± 2.5Official leaderboard, measured May 2026Configuration: Pass^3 18.0
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
    Claude Opus 4.7 29.3 ± 2.5 37.9 18.0 16.2
    Every result from this document

    Link to this row

  6. 16GPT-5.4OpenAI27.7
    Printed as 27.7 ± 1.5Official leaderboard, measured May 2026
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
    GPT-5.4 27.7 ± 1.5 37.7 13.0 39.8
    Every result from this document

    Link to this row

  7. 17Claude Sonnet 5.5 (low)Anthropic · Anthropic harness27.2
    Printed as 27.2%Vendor-reported, measured Sep 2026Configuration: Anthropic harness; 100 tasks; pass@1; low effort; Opus 5 rubric grader; safety classifiers enabled
    Claude Sonnet 5.5 System Card system card, Anthropic, 28 Sep 2026. p. 138, Figure 8.15.B, right panel PhysicianBench (pass@1); orange Sonnet 5.5 low label (visually read printed labels).
    Sonnet 5.5 | PhysicianBench (pass@1) | low 27.2%
    Every result from this document

    Link to this row

  8. 18Claude Haiku 5.5 (medium)Anthropic · Anthropic harness25.2
    Printed as 25.2%Vendor-reported, measured Oct 2026Configuration: Anthropic harness; 100 tasks; pass@1; medium effort (API default); Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 5
    Claude Haiku 5.5 System Card system card, Anthropic, 7 Oct 2026. p. 133, Section 8.11.3; p. 134, Figure 8.11.3.A PhysicianBench panel
    On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
    Every result from this document

    Link to this row

  9. 19Claude Sonnet 4.6Anthropic23.0
    Printed as 23.0 ± 2.6Official leaderboard, measured May 2026
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
    Claude Sonnet 4.6 23.0 ± 2.6 33.2 9.0 22.3
    Every result from this document

    Link to this row

  10. 20DeepSeek V4-ProDeepSeek · Pass@1 over 3 runs18.7
    Printed as 18.7 ± 2.9Official leaderboard, measured May 2026Configuration: Pass@1 over 3 runs; Pass^3 6.0; shared FHIR tool harness; up to 100 turns; high reasoning when supported
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. arXiv HTML 2605.02240v1, Section 5.2, Table 2; DeepSeek V4-Pro row, Pass@1 column (PDF p. 8).
    Model | Pass@1 | Pass@3 | Pass^3 | #Turns DeepSeek V4-Pro | 18.7 ± 2.9 | 27.9 | 6.0 | 35.3
    Every result from this document

    Link to this row

  11. 21Claude Haiku 5.5 (low)Anthropic · Anthropic harness17.8
    Printed as 17.8%Vendor-reported, measured Oct 2026Configuration: Anthropic harness; 100 tasks; pass@1; low effort; Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 5
    Claude Haiku 5.5 System Card system card, Anthropic, 7 Oct 2026. p. 133, Section 8.11.3; p. 134, Figure 8.11.3.A PhysicianBench panel
    On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
    Every result from this document

    Link to this row

  12. 22Kimi-K2.6Moonshot AI · open source17.0
    Printed as 17.0 ± 2.6Official leaderboard, measured May 2026Configuration: open source
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
    Kimi-K2.6 17.0 ± 2.6 26.3 5.0 42.4
    Every result from this document

    Link to this row

  13. 23MiMo-v2.5-ProXiaomi · Pass@1 over 3 runs16.7
    Printed as 16.7 ± 4.0Official leaderboard, measured May 2026Configuration: Pass@1 over 3 runs; Pass^3 6.0; shared FHIR tool harness; up to 100 turns; high reasoning when supported
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. arXiv HTML 2605.02240v1, Section 5.2, Table 2; MiMo-v2.5-Pro row, Pass@1 column (PDF p. 8).
    Model | Pass@1 | Pass@3 | Pass^3 | #Turns MiMo-v2.5-Pro | 16.7 ± 4.0 | 23.6 | 6.0 | 29.5
    Every result from this document

    Link to this row

  14. 24Qwen3.6-PlusAlibaba13.7
    Printed as 13.7 ± 4.0Official leaderboard, measured May 2026
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
    Qwen3.6-Plus 13.7 ± 4.0 22.6 2.0 28.0
    Every result from this document

    Link to this row

  15. 25MiniMax M2.7MiniMax · Pass@1 over 3 runs8.7
    Printed as 8.7 ± 1.2Official leaderboard, measured May 2026Configuration: Pass@1 over 3 runs; Pass^3 1.0; shared FHIR tool harness; up to 100 turns; high reasoning when supported
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. arXiv HTML 2605.02240v1, Section 5.2, Table 2; MiniMax M2.7 row, Pass@1 column (PDF p. 8).
    Model | Pass@1 | Pass@3 | Pass^3 | #Turns MiniMax M2.7 | 8.7 ± 1.2 | 15.9 | 1.0 | 29.7
    Every result from this document

    Link to this row

  16. 26Gemini Pro 3.1Google6.0
    Printed as 6.0 ± 1.0Official leaderboard, measured May 2026
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
    Gemini Pro 3.1 6.0 ± 1.0 9.3 3.0 30.4
    Every result from this document

    Link to this row

  17. 27Grok-4.20SpaceX AI5.3
    Printed as 5.3 ± 3.2Official leaderboard, measured May 2026
    PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. p. 8, Table 2 (Proprietary Models block)
    Grok-4.20 5.3 ± 3.2 9.7 1.0 16.7
    Every result from this document

    Link to this row

  18. 28Claude Haiku 4.5 (extended thinking)Anthropic · Anthropic harness1.4
    Printed as 1.4%Vendor-reported, measured Oct 2026Configuration: Anthropic harness; 100 tasks; pass@1; extended thinking (no effort setting); Opus 5 rubric grader; safety classifiers enabled; measured for the Haiku 5.5 card
    Claude Haiku 5.5 System Card system card, Anthropic, 7 Oct 2026. p. 132, Figure 8.11.2.A PhysicianBench pass rate, Claude Haiku 4.5 bar label (visually read printed label; not repeated in the text)
    Claude Haiku 4.5 | 1.4%
    Every result from this document

    Link to this row

Pass@1 success rate %, higher better (3 independent runs; Pass^3 also reported). Bar colour: the lab; a pale bar with an outline is a number the model’s own lab reported. Select a row for its quote, page and document.

Documents

4 documents support this board