DischargeBench: Pre-Deployment Evaluation of LLM Discharge Educators through Persona-Grounded Patient Simulation
DischargeBench, a pre-deployment benchmark in which a candidate language model teaches a persona-grounded simulated patient in an open-ended, multi-turn discharge-education conversation. The benchmark (MIMIC-IV-Ext-DischargeBench) has 320 cases across 16 ICD chapters, each paired with a persona varying personality, education level, health literacy and recall. Conversations are scored on conversation quality, a topic checklist, simulated patient comprehension and a claim-level safety audit by a language-model judge validated against physician annotations.
Publisher
arXiv (University of Massachusetts Lowell; University of Massachusetts Amherst)
Published
22 Jul 2026
Added
1 month ago
DOI
—
Key Findings
- 320 MIMIC-IV cases over 16 ICD chapters with four persona axes (v2; v1 had 477 cases over 24 chapters)
- Across eight closed- and open-source models, the best model scored 0.70 on the topic checklist and 0.72 on simulated comprehension
- Distrustful and anxious simulated patients lowered comprehension for every model
- The authors present the benchmark as a screening step before human evaluation, not as evidence of readiness to educate patients
Methodology Notes
Simulation-based benchmark with model-simulated patients and a model judge validated against physician annotations. v1 submitted 22 July 2026 and announced 21 September 2026 (the published_date is the v1 submission date); v2 posted 13 October 2026 retitled the paper, reduced the case set from 477 to 320 and replaced the factual-consistency axis with a Claim-Level Safety Audit. CC BY. The underlying notes require MIMIC-IV credentialed access.
Sources
arXiv preprint(opens in a new tab) (primary)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Jang, Won Seok, Yao, Zonghai, Yu, Hong
Tags
Cite This
APA
Jang, Won Seok, Yao, Zonghai, Yu, Hong. (2026). DischargeBench: Pre-Deployment Evaluation of LLM Discharge Educators through Persona-Grounded Patient Simulation. arXiv (University of Massachusetts Lowell; University of Massachusetts Amherst). https://arxiv.org/abs/2609.20827
Related Insights
API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
arXiv (Stanford University) · 8 Sept 2026
Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions
arXiv (Thomas Lord Department of Computer Science and Keck School of Medicine, University of Southern California) · 15 Apr 2025
Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations
arXiv (accepted to Machine Learning for Healthcare, MLHC 2026); Northeastern University · 14 Jul 2026