MedHELM
Holistic clinical evaluation across 121 tasks in a clinician-validated taxonomy, ranked by mean win rate.
Stanford CRFMOfficial pagePaper
More about this boardLess
Holistic evaluation of LLMs on 121 clinical tasks across 5 categories and 22 subcategories (31 datasets) in a clinician-validated taxonomy; ranked by mean win rate.
Version 5.0.0, last updated May 14, 2026, run by the Stanford-led maintainers on a roughly quarterly cadence. No Claude 5 family or GPT-5.6 rows yet. Mean win rate is relative to the evaluated cohort, so scores shift whenever the model set changes.
Published by Stanford CRFM / HAI and multi-institution collaborators, released Feb 2025. 121 tasks / 31 datasets. Mean win rate 0-1, higher better.
- Pairwise win rate. Nature Medicine 2026 final paper, Table 1; 37 benchmarks and nine historical models. Unit: proportion.
- Headline metric
- Mean win rate
- Version
- v5.0.0
- Version released
- 8 May 2026
- Board last updated
- 14 May 2026
- A newer version exists
- No
- Paper version
- Final journal paper: 37-benchmark snapshot
- Models on the board
- 11
- Rows
- 10
- Models
- 10 of 11on the board
- Labs
- 5
- Last measured
- May 2026
- Leader
- 0.652Gemini 3.1 Pro (Preview)
Ranking
Mean win rate 0-1, higher better
- Anthropic
- OpenAI
- Meta
- Other labs
All official leaderboard
10 of 10 rows
Rows and sources
Open a row for the quote, the page and the document.
1Gemini 3.1 Pro (Preview) 0.652
Printed as 0.652Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')1 Gemini 3.1 Pro (Preview) Google 0.652
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.6520833333333333, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
2Gemini 3.5 Flash 0.642
Printed as 0.642Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')2 Gemini-3.5-flash Google 0.642
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.6416666666666667, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
3Muse Spark (2026-04-08) 0.621
Printed as 0.621Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')3 Muse Spark (2026-04-08) Meta 0.621
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.6208333333333333, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
4GPT-5.4 mini 0.552
Printed as 0.552Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')4 GPT-5.4 mini OpenAI 0.552
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.5520833333333334, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
5GPT-5.4 (2026-03-05) 0.538
Printed as 0.538Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')5 GPT-5.4 (2026-03-05) OpenAI 0.538
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.5375, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
6Gemini 2.5 Pro 0.529
Printed as 0.529Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')6 Gemini 2.5 Pro (05-06 preview) Google 0.529
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.5291666666666667, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
7DeepSeek R1 0.485
Printed as 0.485Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')7 DeepSeek R1 DeepSeek 0.485
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.48541666666666666, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
8Claude 4.6 Opus 0.456
Printed as 0.456Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')8 Claude 4.6 Opus Anthropic 0.456
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.45625, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
9Claude 3.7 Sonnet 0.450
Printed as 0.45Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')9 Claude 3.7 Sonnet (20250219) Anthropic 0.45
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.45, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
10Gemini 2.0 Flash 0.342
Printed as 0.342Official leaderboard, measured May 2026MedHELM leaderboard (medhelm.org), v5.0.0 official leaderboard, Stanford CRFM (MedHELM), 14 May 2026. medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')10 Gemini 2.0 Flash Google 0.342
Every result from this document- Also printed in MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate): 0.3416666666666667, releases/v5.0.0/groups/medhelm_scenarios.json, table 'Accuracy', column 'Mean win rate'
Documents
2
- MedHELM leaderboard (medhelm.org), v5.0.0official leaderboard, Stanford CRFM (MedHELM), 14 May 2026Results it supports
- MedHELM v5.0.0 release data: medhelm_scenarios group table (mean win rate)official leaderboard, Stanford CRFM (MedHELM), 8 May 2026Results it supports