AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment

Key Findings

• — 914 cases: 697 consensus cases and 217 physician-confirmed ambiguous cases; four acuity levels
• — Clear-case exact-match accuracy in QA format ranged from 53.3% (Llama 3.3 70B) to 85.3% (Claude Opus 4.7), mean 70.6% across 12 models
• — Conversational replies reduced over-triage but increased under-triage relative to QA, especially for higher-acuity cases
• — Lowest-acuity (A-level) error rates ranged from 13% to 84.4% across models
• — On ambiguous cases no model matched the spread of physician judgments; model predictions were more concentrated than expert uncertainty

Methodology Notes

Five public source datasets relabelled to a shared four-level framework by emergency physicians; five samples per case, modal label with ties toward higher acuity; conversational judge validated against two blinded human annotators (judge-human agreement 74.2% vs human-human 64.4% over 132 responses). Models: GPT-5.4, GPT-5-mini, GPT-4.1, Claude Opus 4.7, Sonnet 4.6, Haiku 4.5, Gemini 2.5 Pro and Flash, plus open-weight Llama 3.3 70B, Qwen 2.5 72B and others. v1 posted 2026-05-12; v2 2026-09-25 (41 pages) states it is under review for the NeurIPS 2026 Evaluations and Datasets track. Verified from arXiv abs (200) and HTML render (200).

Sources