Clinical Benchmarks

PhysicianBench

Agents carrying out long-horizon physician workflows inside real EHR systems, verified by execution against those systems.

More about this boardLess

LLM agents on long-horizon composite physician workflows inside real EHR environments, with execution-grounded verification against actual EHR systems via standard commercial APIs.

Scores come from the paper; there is no standalone public leaderboard, and the results also feed the ARISE MAST composite. The lower half of the table falls steeply, with several agents near 1 percent on Pass^3 consistency.

Published by Academic team (Ruoqi Liu, Imran Q. Mohiuddin et al., arXiv 2605.02240); also a MAST component, released May 2026. 100 real-world clinical tasks, 21 specialties, 670 structured checkpoints (~27 tool calls per task). Pass@1 success rate %, higher better (3 independent runs; Pass^3 also reported).

Table
Table 2, p. 8
Headline metric
Pass@1 (%), mean ± sd over 3 runs
Paper date
4 May 2026
Paper version
v1
Models in the paper's table
12
Publisher
Academic team (Ruoqi Liu, Imran Q. Mohiuddin et al., arXiv 2605.02240); also a MAST component
Rows
28
Models
20 on the board
Released
May 2026
Measured to
Oct 2026
Unit
100 clinical tasks
Documents cited
3

Every result on PhysicianBench, on the board's own scale

28 results, one column each, highest first. Pass@1 success rate %, higher better (3 independent runs; Pass^3 also reported).

Column colour is the model's lab. The table below says who ran each result.

Every published result, ranked on this board alone

28 rows. Higher is better. Bars run from 0 to 100; whiskers show a published interval. Select a row for its quote, page and document.

1Claude Opus 5.5 (max)Anthropic · Anthropic harness68.4Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
As printed
68.4%
Where
pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Opus 5.5 (max).
Setting
Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
2Claude Sonnet 5.5 (max)Anthropic · Anthropic harness63.2Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
As printed
63.2%
Where
pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Sonnet 5.5 (max).
Setting
Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
3Claude Fable 5.1 (max)Anthropic · Anthropic harness61.0Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
As printed
61.0%
Where
pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Fable 5.1 (max).
Setting
Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
4Claude Opus 5 (max)Anthropic · Anthropic harness57.6Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
As printed
57.6%
Where
pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Opus 5 (max).
Setting
Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
5Claude Sonnet 5.5 (xhigh)Anthropic · Anthropic harness56.4Model's own labSep 2026[1]
Sonnet 5.5 | PhysicianBench (pass@1) | xhigh 56.4%
As printed
56.4%
Where
p. 138, Figure 8.15.B, right panel PhysicianBench (pass@1); orange Sonnet 5.5 xhigh label (visually read printed labels).
Setting
Anthropic harness; 100 tasks; pass@1; xhigh effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
6Claude Sonnet 5.5 (high)Anthropic · Anthropic harness47.6Model's own labSep 2026[1]
Sonnet 5.5 | PhysicianBench (pass@1) | high 47.6%
As printed
47.6%
Where
p. 138, Figure 8.15.B, right panel PhysicianBench (pass@1); orange Sonnet 5.5 high label (visually read printed labels).
Setting
Anthropic harness; 100 tasks; pass@1; high effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
7GPT-5.5OpenAI · pass@146.3Board publisherMay 2026[2]
GPT-5.5 46.3 ± 1.2 57.4 28.0 41.9
As printed
46.3 ± 1.2
Where
p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
Setting
pass@1; Pass^3 28.0
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
8Claude Haiku 5.5 (max)Anthropic · Anthropic harness43.0Model's own labOct 2026[3]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an electronic health record (EHR). Claude Haiku 5.5 passed 43.0% of attempts (five attempts on each task).
As printed
43.0%
Where
pp. 132-133, Section 8.11.2, Figure 8.11.2.A caption and paragraph; also p. 133, Section 8.11.3
Setting
Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 1 and 2
Reported by
vendor-reported
Document
[3] Claude Haiku 5.5 System Card
9Claude Sonnet 5 (max)Anthropic · Anthropic harness37.4Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
As printed
37.4%
Where
pp. 138–139, Section 8.15.3; PhysicianBench pass@1, Claude Sonnet 5 (max).
Setting
Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
10Claude Haiku 5.5 (xhigh)Anthropic · Anthropic harness35.8Model's own labOct 2026[3]
On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
As printed
35.8%
Where
p. 133, Section 8.11.3; p. 134, Figure 8.11.3.A PhysicianBench panel
Setting
Anthropic harness; 100 tasks; pass@1; xhigh effort; Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 5
Reported by
vendor-reported
Document
[3] Claude Haiku 5.5 System Card
11GPT-6 Luna (Anthropic run)OpenAI · Run by Anthropic through…33.6Independent runOct 2026[3]
This was 9.4 points above GPT-6 Luna at 33.6% (95% confidence interval 2.6 to 16.2 points, paired by task)
As printed
33.6%
Where
p. 133, Section 8.11.2; p. 132, Figure 8.11.2.A GPT-6 Luna bar; p. 131-132, Section 8.11.1 says GPT-6 Luna scores come from Anthropic's own runs
Setting
Run by Anthropic through OpenAI's public API; Anthropic harness; 100 tasks; pass@1; max effort; Opus 5 rubric grader; not OpenAI's published figure
Reported by
independent run
Document
[3] Claude Haiku 5.5 System Card
12Claude Opus 4.6Anthropic · Pass^3 18.031.7Board publisherMay 2026[2]
Claude Opus 4.6 31.7 ± 2.3 41.5 18.0 25.2
As printed
31.7 ± 2.3
Where
p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
Setting
Pass^3 18.0
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
13Claude Haiku 5.5 (high)Anthropic · Anthropic harness31.6Model's own labOct 2026[3]
On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
As printed
31.6%
Where
p. 133, Section 8.11.3; p. 134, Figure 8.11.3.A PhysicianBench panel
Setting
Anthropic harness; 100 tasks; pass@1; high effort; Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 5
Reported by
vendor-reported
Document
[3] Claude Haiku 5.5 System Card
14Claude Sonnet 5.5 (medium)Anthropic · Anthropic harness30.0Model's own labSep 2026[1]
Sonnet 5.5 | PhysicianBench (pass@1) | medium 30.0%
As printed
30.0%
Where
p. 138, Figure 8.15.B, right panel PhysicianBench (pass@1); orange Sonnet 5.5 medium label (visually read printed labels).
Setting
Anthropic harness; 100 tasks; pass@1; medium effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
15Claude Opus 4.7Anthropic · Pass^3 18.029.3Board publisherMay 2026[2]
Claude Opus 4.7 29.3 ± 2.5 37.9 18.0 16.2
As printed
29.3 ± 2.5
Where
p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
Setting
Pass^3 18.0
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
16GPT-5.4OpenAI27.7Board publisherMay 2026[2]
GPT-5.4 27.7 ± 1.5 37.7 13.0 39.8
As printed
27.7 ± 1.5
Where
p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
17Claude Sonnet 5.5 (low)Anthropic · Anthropic harness27.2Model's own labSep 2026[1]
Sonnet 5.5 | PhysicianBench (pass@1) | low 27.2%
As printed
27.2%
Where
p. 138, Figure 8.15.B, right panel PhysicianBench (pass@1); orange Sonnet 5.5 low label (visually read printed labels).
Setting
Anthropic harness; 100 tasks; pass@1; low effort; Opus 5 rubric grader; safety classifiers enabled
Reported by
vendor-reported
Document
[1] Claude Sonnet 5.5 System Card
18Claude Haiku 5.5 (medium)Anthropic · Anthropic harness25.2Model's own labOct 2026[3]
On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
As printed
25.2%
Where
p. 133, Section 8.11.3; p. 134, Figure 8.11.3.A PhysicianBench panel
Setting
Anthropic harness; 100 tasks; pass@1; medium effort (API default); Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 5
Reported by
vendor-reported
Document
[3] Claude Haiku 5.5 System Card
19Claude Sonnet 4.6Anthropic23.0Board publisherMay 2026[2]
Claude Sonnet 4.6 23.0 ± 2.6 33.2 9.0 22.3
As printed
23.0 ± 2.6
Where
p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
20DeepSeek V4-ProDeepSeek · Pass@1 over 3 runs18.7Board publisherMay 2026[2]
Model | Pass@1 | Pass@3 | Pass^3 | #Turns DeepSeek V4-Pro | 18.7 ± 2.9 | 27.9 | 6.0 | 35.3
As printed
18.7 ± 2.9
Where
arXiv HTML 2605.02240v1, Section 5.2, Table 2; DeepSeek V4-Pro row, Pass@1 column (PDF p. 8).
Setting
Pass@1 over 3 runs; Pass^3 6.0; shared FHIR tool harness; up to 100 turns; high reasoning when supported
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
21Claude Haiku 5.5 (low)Anthropic · Anthropic harness17.8Model's own labOct 2026[3]
On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
As printed
17.8%
Where
p. 133, Section 8.11.3; p. 134, Figure 8.11.3.A PhysicianBench panel
Setting
Anthropic harness; 100 tasks; pass@1; low effort; Opus 5 rubric grader; safety classifiers enabled; five attempts per task; run October 5
Reported by
vendor-reported
Document
[3] Claude Haiku 5.5 System Card
22Kimi-K2.6Moonshot AI · open source17.0Board publisherMay 2026[2]
Kimi-K2.6 17.0 ± 2.6 26.3 5.0 42.4
As printed
17.0 ± 2.6
Where
p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
Setting
open source
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
23MiMo-v2.5-ProXiaomi · Pass@1 over 3 runs16.7Board publisherMay 2026[2]
Model | Pass@1 | Pass@3 | Pass^3 | #Turns MiMo-v2.5-Pro | 16.7 ± 4.0 | 23.6 | 6.0 | 29.5
As printed
16.7 ± 4.0
Where
arXiv HTML 2605.02240v1, Section 5.2, Table 2; MiMo-v2.5-Pro row, Pass@1 column (PDF p. 8).
Setting
Pass@1 over 3 runs; Pass^3 6.0; shared FHIR tool harness; up to 100 turns; high reasoning when supported
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
24Qwen3.6-PlusAlibaba13.7Board publisherMay 2026[2]
Qwen3.6-Plus 13.7 ± 4.0 22.6 2.0 28.0
As printed
13.7 ± 4.0
Where
p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
25MiniMax M2.7MiniMax · Pass@1 over 3 runs8.7Board publisherMay 2026[2]
Model | Pass@1 | Pass@3 | Pass^3 | #Turns MiniMax M2.7 | 8.7 ± 1.2 | 15.9 | 1.0 | 29.7
As printed
8.7 ± 1.2
Where
arXiv HTML 2605.02240v1, Section 5.2, Table 2; MiniMax M2.7 row, Pass@1 column (PDF p. 8).
Setting
Pass@1 over 3 runs; Pass^3 1.0; shared FHIR tool harness; up to 100 turns; high reasoning when supported
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
26Gemini Pro 3.1Google6.0Board publisherMay 2026[2]
Gemini Pro 3.1 6.0 ± 1.0 9.3 3.0 30.4
As printed
6.0 ± 1.0
Where
p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
27Grok-4.20SpaceX AI5.3Board publisherMay 2026[2]
Grok-4.20 5.3 ± 3.2 9.7 1.0 16.7
As printed
5.3 ± 3.2
Where
p. 8, Table 2 (Proprietary Models block)
Reported by
official leaderboard
Document
[2] PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)
28Claude Haiku 4.5 (extended thinking)Anthropic · Anthropic harness1.4Model's own labOct 2026[3]
Claude Haiku 4.5 | 1.4%
As printed
1.4%
Where
p. 132, Figure 8.11.2.A PhysicianBench pass rate, Claude Haiku 4.5 bar label (visually read printed label; not repeated in the text)
Setting
Anthropic harness; 100 tasks; pass@1; extended thinking (no effort setting); Opus 5 rubric grader; safety classifiers enabled; measured for the Haiku 5.5 card
Reported by
vendor-reported
Document
[3] Claude Haiku 5.5 System Card
References

3 documents support this board

  1. 1.Claude Sonnet 5.5 System Cardsystem card, Anthropic, 28 Sep 2026. Every result it supports
  2. 2.PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. Every result it supports
  3. 3.Claude Haiku 5.5 System Cardsystem card, Anthropic, 7 Oct 2026. Every result it supports