Between Help and Harm: An Evaluation Study of Mental Health Crisis Handling by Large Language Models
Peer-reviewed study introducing a taxonomy of six clinically-informed mental-health crisis categories, an evaluation dataset of over 2,000 user inputs drawn from twelve public conversational datasets, and an expert protocol for rating response appropriateness and safety. Assesses how leading LLMs handle crisis conversations.
Key Findings
- Releases a six-category crisis taxonomy and an evaluation dataset of 2,000+ inputs aggregated from twelve datasets
- A non-negligible share of LLM responses were inappropriate or harmful, worst for self-harm and suicidal ideation
- Safety failures tracked alignment quality more than raw model scale, and models were weakest on indirect distress signals
Methodology Notes
Peer-reviewed, JMIR Mental Health 2026, vol. 13, article e88435 (11 June 2026), DOI 10.2196/88435. Published successor to arXiv preprint 2509.24857. Benchmark built by aggregating twelve existing datasets; expert-rated response protocol. mental.jmir.org is JS-rendered to fetchers; DOI/date confirmed via Crossref.
Sources
JMIR Mental Health article (primary)
Authors
Adrian Arnaiz-Rodriguez, Miguel Baidal, Erik Derner, Jenn Layton Annable, Mark Ball, Mark Ince, Elvira Perez Vallejos, Nuria Oliver
Tags
Cite This
APA
Adrian Arnaiz-Rodriguez et al. (2026). Between Help and Harm: An Evaluation Study of Mental Health Crisis Handling by Large Language Models. JMIR Mental Health. https://mental.jmir.org/2026/1/e88435
Related Insights
Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs
arXiv (ELLIS Alicante-led) · 29 Sept 2025
CRADLE Bench: A Clinician-Annotated Benchmark for Multi-Faceted Mental Health Crisis and Safety Risk Detection
arXiv (Emory University-led) · 27 Oct 2025
VERA-MH: Reliability and Validity of an Open-Source AI Safety Evaluation in Mental Health
arXiv (Spring Health / Slingshot AI-affiliated author team) · 4 Feb 2026
Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment
Psychiatric Services (American Psychiatric Association); RAND-led author team · 26 Aug 2025