CHI-Bench
Long-horizon healthcare operations workflows for agents: prior authorization, utilization management, and care management.
More about this boardLess
Long-horizon US healthcare operations workflows for agents (prior authorization, utilization management, care management), 60-80 step tasks across 4-6 stages, judged by deterministic unit tests plus an LLM judge for evidence grounding, consent, and cross-stage consistency.
Released May 20, 2026; updated August 12, 2026 across 45 harness configurations. The launch report led with reliability, not capability: no agent stayed above 20 percent across three identical runs. Harness choice matters as much as model choice, and the board accepts community submissions, so rows mix author-run and submitted results.
Published by actAVA.ai, released May 2026. 75 workflows (25 per domain), 21 healthcare applications, 200+ MCP tools. Pass@1 with binary 0/1 reward, higher better.
- Headline metric
- pass@1 accuracy, 75 tasks (25 each: prior authorization, utilization management, care management)
- Rows on the board
- 45
- Board last updated
- 12 Aug 2026
- Board version
- CHI-Bench v1.0.0
- Publisher
- actAVA.ai
- Rows
- 43
- Models
- 25 of 45 on the board
- Released
- May 2026
- Measured to
- Aug 2026
- Unit
- 75 operations workflows
- Documents cited
- 1
Every result on CHI-Bench, on the board's own scale
43 results, one column each, highest first. Pass@1 with binary 0/1 reward, higher better.
- erius + claude-opus-5
- claude-code + claude-opus-5
- erius + claude-opus-4-8
- claude-code + claude-opus-4-8
- claude-code + claude-opus-4-6
- claude-code + claude-sonnet-4-6
- codex + gpt-5.6-sol
- openai-agents + kimi-k3
- claude-code + claude-opus-4-7
- claude-code + claude-fable-5
- hermes + MedGuard
- codex + gpt-5.5
- claude-code + claude-sonnet-5
- hermes + glm-5.1
- openai-agents + glm-5.1
- openai-agents + glm-5.2
- openclaw + claude-opus-4-7
- openclaw + glm-5.1
- hermes + qwen-3.6-max
- codex + gpt-5.4
- hermes + kimi-k2.6
- openai-agents + qwen-3.6-max
- openai-agents + kimi-k2.6
- openai-agents + deepseek-v4-pro
- hermes + deepseek-v4-pro
Column colour is the model's lab. The table below says who ran each result.
Every published result, ranked on this board alone
43 rows. Higher is better. Bars run from 0 to 100; whiskers show a published interval. Select a row for its quote, page and document.
1erius + claude-opus-5Humana (harness) / Anthropic (model) · community-submitted harness…54.7Independent runAug 2026[1]
01 | erius submitted by Michael Johnson (MJ) | claude-opus-5 | Proprietary | 54.7% | 72.0% | 36.0% | 56.0% | 2026-07-26
2claude-code + claude-opus-5Anthropic37.3Board publisherJul 2026[1]
03 | claude-code | claude-opus-5 | Proprietary | 37.3% | 20.0% | 32.0% | 60.0% | 2026-07-24
2erius + claude-opus-4-8Humana / Anthropic37.3Independent runAug 2026[1]
02 | erius submitted by Michael Johnson (MJ) | claude-opus-4-8 | Proprietary | 37.3% | 40.0% | 16.0% | 56.0% | 2026-06-05
4claude-code + claude-opus-4-8Anthropic33.3Board publisherMay 2026[1]
04 | claude-code | claude-opus-4-8 | Proprietary | 33.3% | 32.0% | 28.0% | 40.0% | 2026-05-28
5claude-code + claude-opus-4-6Anthropic28.0Board publisherMay 2026[1]
05 | claude-code | claude-opus-4-6 | Proprietary | 28.0% | 20.0% | 36.0% | 28.0% | 2026-05-01
6claude-code + claude-sonnet-4-6Anthropic26.2Board publisherApr 2026[1]
06 | claude-code | claude-sonnet-4-6 | Proprietary | 26.2% | 24.0% | 34.7% | 20.0% | 2026-05-01
7codex + gpt-5.6-solOpenAI25.3Board publisherJul 2026[1]
07 | codex | gpt-5.6-sol | Proprietary | 25.3% | 36.0% | 28.0% | 12.0% | 2026-07-24
7openai-agents + kimi-k3Moonshot AI25.3Board publisherJul 2026[1]
08 | openai-agents | kimi-k3 | Open-source | 25.3% | 28.0% | 32.0% | 16.0% | 2026-07-24
9claude-code + claude-opus-4-7Anthropic24.4Board publisherApr 2026[1]
09 | claude-code | claude-opus-4-7 | Proprietary | 24.4% | 24.0% | 17.3% | 32.0% | 2026-05-01
10claude-code + claude-fable-5Anthropic24.0Board publisherJul 2026[1]
10 | claude-code | claude-fable-5 | Proprietary | 24.0% | 24.0% | 24.0% | 24.0% | 2026-07-22
11hermes + MedGuardMedGuard · All Domains pass@122.7Independent runJul 2026[1]
11 | hermes submitted by cuilinke | MedGuard | Open-source | 22.7% | 4.0% | 4.0% | 60.0% | 2026-07-06
12codex + gpt-5.5OpenAI20.9Board publisherApr 2026[1]
12 | codex | gpt-5.5 | Proprietary | 20.9% | 29.3% | 32.0% | 1.3% | 2026-05-01
13claude-code + claude-sonnet-5Anthropic20.0Board publisherJul 2026[1]
13 | claude-code | claude-sonnet-5 | Proprietary | 20.0% | 24.0% | 24.0% | 12.0% | 2026-07-06
14hermes + glm-5.1Zhipu · All Domains pass@118.7Board publisherMay 2026[1]
15 | hermes | glm-5.1 | Open-source | 18.7% | 10.7% | 34.7% | 10.7% | 2026-05-01
14openai-agents + glm-5.1Zhipu · All Domains pass@118.7Board publisherMay 2026[1]
14 | openai-agents | glm-5.1 | Open-source | 18.7% | 18.7% | 33.3% | 4.0% | 2026-05-01
14openai-agents + glm-5.2Zhipu · All Domains pass@118.7Board publisherJul 2026[1]
16 | openai-agents | glm-5.2 | Open-source | 18.7% | 20.0% | 32.0% | 4.0% | 2026-07-06
17openclaw + claude-opus-4-7Anthropic · All Domains pass@117.3Board publisherMay 2026[1]
17 | openclaw | claude-opus-4-7 | Proprietary | 17.3% | 18.7% | 13.3% | 20.0% | 2026-05-01
18openclaw + glm-5.1Zhipu · All Domains pass@116.9Board publisherMay 2026[1]
18 | openclaw | glm-5.1 | Open-source | 16.9% | 13.3% | 26.7% | 10.7% | 2026-05-01
19hermes + qwen-3.6-maxAlibaba · All Domains pass@116.4Board publisherMay 2026[1]
19 | hermes | qwen-3.6-max | Open-source | 16.4% | 9.3% | 26.7% | 13.3% | 2026-05-01
20codex + gpt-5.4OpenAI · All Domains pass@116.0Board publisherApr 2026[1]
20 | codex | gpt-5.4 | Proprietary | 16.0% | 24.0% | 17.3% | 6.7% | 2026-05-01
21hermes + kimi-k2.6Moonshot AI · All Domains pass@115.6Board publisherMay 2026[1]
22 | hermes | kimi-k2.6 | Open-source | 15.6% | 18.7% | 21.3% | 6.7% | 2026-05-01
21openai-agents + qwen-3.6-maxAlibaba · All Domains pass@115.6Board publisherMay 2026[1]
21 | openai-agents | qwen-3.6-max | Open-source | 15.6% | 16.0% | 26.7% | 4.0% | 2026-05-01
23openai-agents + kimi-k2.6Moonshot AI · All Domains pass@115.1Board publisherMay 2026[1]
23 | openai-agents | kimi-k2.6 | Open-source | 15.1% | 17.3% | 25.3% | 2.7% | 2026-05-01
24openai-agents + deepseek-v4-proDeepSeek · All Domains pass@114.2Board publisherMay 2026[1]
24 | openai-agents | deepseek-v4-pro | Open-source | 14.2% | 10.7% | 28.0% | 4.0% | 2026-05-01
25hermes + deepseek-v4-proDeepSeek · All Domains pass@113.8Board publisherMay 2026[1]
25 | hermes | deepseek-v4-pro | Open-source | 13.8% | 8.0% | 25.3% | 8.0% | 2026-05-01
26codex + gpt-5.6-lunaOpenAI · All Domains pass@113.3Board publisherJul 2026[1]
27 | codex | gpt-5.6-luna | Proprietary | 13.3% | 20.0% | 16.0% | 4.0% | 2026-07-24
26codex + gpt-5.6-terraOpenAI · All Domains pass@113.3Board publisherJul 2026[1]
26 | codex | gpt-5.6-terra | Proprietary | 13.3% | 12.0% | 20.0% | 8.0% | 2026-07-24
28gemini-cli + gemini-3-flashGoogle · All Domains pass@112.5Board publisherApr 2026[1]
28 | gemini-cli | gemini-3-flash | Proprietary | 12.5% | 18.7% | 18.7% | 0.0% | 2026-05-01
29deepagents + glm-5.1Zhipu · All Domains pass@111.1Board publisherMay 2026[1]
30 | deepagents | glm-5.1 | Open-source | 11.1% | 17.3% | 10.7% | 5.3% | 2026-05-01
29openclaw + deepseek-v4-proDeepSeek · All Domains pass@111.1Board publisherMay 2026[1]
29 | openclaw | deepseek-v4-pro | Open-source | 11.1% | 14.7% | 12.0% | 6.7% | 2026-05-01
31deepagents + deepseek-v4-proDeepSeek · All Domains pass@110.7Board publisherMay 2026[1]
31 | deepagents | deepseek-v4-pro | Open-source | 10.7% | 14.7% | 10.7% | 6.7% | 2026-05-01
32openclaw + kimi-k2.6Moonshot AI · All Domains pass@110.2Board publisherMay 2026[1]
32 | openclaw | kimi-k2.6 | Open-source | 10.2% | 12.0% | 18.7% | 0.0% | 2026-05-01
33deepagents + qwen-3.6-maxAlibaba · All Domains pass@19.3Board publisherMay 2026[1]
33 | deepagents | qwen-3.6-max | Open-source | 9.3% | 12.0% | 10.7% | 5.3% | 2026-05-01
34codex + gpt-5.4-miniOpenAI · All Domains pass@18.4Board publisherMay 2026[1]
34 | codex | gpt-5.4-mini | Proprietary | 8.4% | 10.7% | 13.3% | 1.3% | 2026-05-01
35openai-agents + TML Inkling 256KThinking Machines · All Domains pass@18.0Board publisherJul 2026[1]
35 | openai-agents | TML Inkling 256K | Open-source | 8.0% | 4.0% | 16.0% | 4.0% | 2026-07-24
36claude-code + claude-haiku-4-5Anthropic · All Domains pass@16.2Board publisherApr 2026[1]
37 | claude-code | claude-haiku-4-5 | Proprietary | 6.2% | 0.0% | 14.7% | 4.0% | 2026-05-01
37openai-agents + grok-4.3SpaceX AI · All Domains pass@15.8Board publisherMay 2026[1]
38 | openai-agents | grok-4.3 | Open-source | 5.8% | 0.0% | 16.0% | 1.3% | 2026-05-01
38openclaw + qwen-3.6-maxAlibaba · All Domains pass@14.9Board publisherMay 2026[1]
39 | openclaw | qwen-3.6-max | Open-source | 4.9% | 10.7% | 4.0% | 0.0% | 2026-05-01
39hermes + grok-4.3SpaceX AI · All Domains pass@14.4Board publisherMay 2026[1]
40 | hermes | grok-4.3 | Open-source | 4.4% | 0.0% | 13.3% | 0.0% | 2026-05-01
40deepagents + kimi-k2.6Moonshot AI · All Domains pass@13.1Board publisherMay 2026[1]
41 | deepagents | kimi-k2.6 | Open-source | 3.1% | 8.0% | 1.3% | 0.0% | 2026-05-01
41deepagents + grok-4.3SpaceX AI · All Domains pass@12.2Board publisherMay 2026[1]
42 | deepagents | grok-4.3 | Open-source | 2.2% | 0.0% | 5.3% | 1.3% | 2026-05-01
42openclaw + grok-4.3SpaceX AI · All Domains pass@10.4Board publisherMay 2026[1]
43 | openclaw | grok-4.3 | Open-source | 0.4% | 1.3% | 0.0% | 0.0% | 2026-05-01
43openai-agents + Nemotron 3 Ultra 256KNVIDIA · All Domains pass@10.0Board publisherJul 2026[1]
44 | openai-agents | Nemotron 3 Ultra 256K | Open-source | 0.0% | 0.0% | 0.0% | 0.0% | 2026-07-24
1 document supports this board
- 1.CHI-Bench leaderboard (actAVA)official leaderboard, actAVA, 12 Aug 2026. Every result it supports