PhysicianBench
Agents carrying out long-horizon physician workflows inside real EHR systems, verified by execution against those systems.
More about this boardLess
LLM agents on long-horizon composite physician workflows inside real EHR environments, with execution-grounded verification against actual EHR systems via standard commercial APIs.
Scores come from the paper; there is no standalone public leaderboard, and the results also feed the ARISE MAST composite. The lower half of the table falls steeply, with several agents near 1 percent on Pass^3 consistency.
Published by Academic team (Ruoqi Liu, Imran Q. Mohiuddin et al., arXiv 2605.02240); also a MAST component, released May 2026. 100 real-world clinical tasks, 21 specialties, 670 structured checkpoints (~27 tool calls per task). Pass@1 success rate %, higher better (3 independent runs; Pass^3 also reported).
- Table
- Table 2, p. 8
- Headline metric
- Pass@1 (%), mean ± sd over 3 runs
- Paper date
- 4 May 2026
- Paper version
- v1
- Models in the paper's table
- 12
- Publisher
- Academic team (Ruoqi Liu, Imran Q. Mohiuddin et al., arXiv 2605.02240); also a MAST component
- Rows
- 28
- Models
- 20 on the board
- Released
- May 2026
- Measured to
- Oct 2026
- Unit
- 100 clinical tasks
- Documents cited
- 3
Every result on PhysicianBench, on the board's own scale
28 results, one column each, highest first. Pass@1 success rate %, higher better (3 independent runs; Pass^3 also reported).
- Claude Opus 5.5 (max)
- Claude Sonnet 5.5 (max)
- Claude Fable 5.1 (max)
- Claude Opus 5 (max)
- Claude Sonnet 5.5 (xhigh)
- Claude Sonnet 5.5 (high)
- GPT-5.5
- Claude Haiku 5.5 (max)
- Claude Sonnet 5 (max)
- Claude Haiku 5.5 (xhigh)
- GPT-6 Luna (Anthropic run)
- Claude Opus 4.6
- Claude Haiku 5.5 (high)
- Claude Sonnet 5.5 (medium)
- Claude Opus 4.7
- GPT-5.4
- Claude Sonnet 5.5 (low)
- Claude Haiku 5.5 (medium)
- Claude Sonnet 4.6
- DeepSeek V4-Pro
- Claude Haiku 5.5 (low)
- Kimi-K2.6
- MiMo-v2.5-Pro
- Qwen3.6-Plus
- MiniMax M2.7
Column colour is the model's lab. The table below says who ran each result.
Every published result, ranked on this board alone
28 rows. Higher is better. Bars run from 0 to 100; whiskers show a published interval. Select a row for its quote, page and document.
1Claude Opus 5.5 (max)Anthropic · Anthropic harness68.4Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
2Claude Sonnet 5.5 (max)Anthropic · Anthropic harness63.2Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
3Claude Fable 5.1 (max)Anthropic · Anthropic harness61.0Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
4Claude Opus 5 (max)Anthropic · Anthropic harness57.6Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
5Claude Sonnet 5.5 (xhigh)Anthropic · Anthropic harness56.4Model's own labSep 2026[1]
Sonnet 5.5 | PhysicianBench (pass@1) | xhigh 56.4%
6Claude Sonnet 5.5 (high)Anthropic · Anthropic harness47.6Model's own labSep 2026[1]
Sonnet 5.5 | PhysicianBench (pass@1) | high 47.6%
7GPT-5.5OpenAI · pass@146.3Board publisherMay 2026[2]
GPT-5.5 46.3 ± 1.2 57.4 28.0 41.9
8Claude Haiku 5.5 (max)Anthropic · Anthropic harness43.0Model's own labOct 2026[3]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an electronic health record (EHR). Claude Haiku 5.5 passed 43.0% of attempts (five attempts on each task).
9Claude Sonnet 5 (max)Anthropic · Anthropic harness37.4Model's own labSep 2026[1]
PhysicianBench is a public benchmark of 100 physician tasks carried out in an EHR. At max effort, Claude Sonnet 5.5 passes 63.2% of attempts, well above Claude Sonnet 5 (37.4%), level with Claude Fable 5.1 (61.0%), about 6 points above Claude Opus 5 (57.6%) and about 5 points below Claude Opus 5.5 (68.4%).
10Claude Haiku 5.5 (xhigh)Anthropic · Anthropic harness35.8Model's own labOct 2026[3]
On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
11GPT-6 Luna (Anthropic run)OpenAI · Run by Anthropic through…33.6Independent runOct 2026[3]
This was 9.4 points above GPT-6 Luna at 33.6% (95% confidence interval 2.6 to 16.2 points, paired by task)
12Claude Opus 4.6Anthropic · Pass^3 18.031.7Board publisherMay 2026[2]
Claude Opus 4.6 31.7 ± 2.3 41.5 18.0 25.2
13Claude Haiku 5.5 (high)Anthropic · Anthropic harness31.6Model's own labOct 2026[3]
On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
14Claude Sonnet 5.5 (medium)Anthropic · Anthropic harness30.0Model's own labSep 2026[1]
Sonnet 5.5 | PhysicianBench (pass@1) | medium 30.0%
15Claude Opus 4.7Anthropic · Pass^3 18.029.3Board publisherMay 2026[2]
Claude Opus 4.7 29.3 ± 2.5 37.9 18.0 16.2
16GPT-5.4OpenAI27.7Board publisherMay 2026[2]
GPT-5.4 27.7 ± 1.5 37.7 13.0 39.8
17Claude Sonnet 5.5 (low)Anthropic · Anthropic harness27.2Model's own labSep 2026[1]
Sonnet 5.5 | PhysicianBench (pass@1) | low 27.2%
18Claude Haiku 5.5 (medium)Anthropic · Anthropic harness25.2Model's own labOct 2026[3]
On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
19Claude Sonnet 4.6Anthropic23.0Board publisherMay 2026[2]
Claude Sonnet 4.6 23.0 ± 2.6 33.2 9.0 22.3
20DeepSeek V4-ProDeepSeek · Pass@1 over 3 runs18.7Board publisherMay 2026[2]
Model | Pass@1 | Pass@3 | Pass^3 | #Turns DeepSeek V4-Pro | 18.7 ± 2.9 | 27.9 | 6.0 | 35.3
21Claude Haiku 5.5 (low)Anthropic · Anthropic harness17.8Model's own labOct 2026[3]
On PhysicianBench (five attempts per task at each level), Haiku 5.5 passed 17.8% of attempts at low, 25.2% at medium, 31.6% at high, 35.8% at xhigh and 43.0% at max.
22Kimi-K2.6Moonshot AI · open source17.0Board publisherMay 2026[2]
Kimi-K2.6 17.0 ± 2.6 26.3 5.0 42.4
23MiMo-v2.5-ProXiaomi · Pass@1 over 3 runs16.7Board publisherMay 2026[2]
Model | Pass@1 | Pass@3 | Pass^3 | #Turns MiMo-v2.5-Pro | 16.7 ± 4.0 | 23.6 | 6.0 | 29.5
24Qwen3.6-PlusAlibaba13.7Board publisherMay 2026[2]
Qwen3.6-Plus 13.7 ± 4.0 22.6 2.0 28.0
25MiniMax M2.7MiniMax · Pass@1 over 3 runs8.7Board publisherMay 2026[2]
Model | Pass@1 | Pass@3 | Pass^3 | #Turns MiniMax M2.7 | 8.7 ± 1.2 | 15.9 | 1.0 | 29.7
26Gemini Pro 3.1Google6.0Board publisherMay 2026[2]
Gemini Pro 3.1 6.0 ± 1.0 9.3 3.0 30.4
27Grok-4.20SpaceX AI5.3Board publisherMay 2026[2]
Grok-4.20 5.3 ± 3.2 9.7 1.0 16.7
28Claude Haiku 4.5 (extended thinking)Anthropic · Anthropic harness1.4Model's own labOct 2026[3]
Claude Haiku 4.5 | 1.4%
3 documents support this board
- 1.Claude Sonnet 5.5 System Cardsystem card, Anthropic, 28 Sep 2026. Every result it supports
- 2.PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF)paper, Stanford University (HealthRex; Liu, Chen et al.), 4 May 2026. Every result it supports
- 3.Claude Haiku 5.5 System Cardsystem card, Anthropic, 7 Oct 2026. Every result it supports