Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs
Introduces a taxonomy of six clinically informed crisis categories and a curated dataset of over 2,200 inputs drawn from twelve mental-health datasets, plus a companion dataset of model responses and evaluations. Five models are assessed on how safely they handle crisis conversations.
Publisher
arXiv (ELLIS Alicante-led)
Published
29 Sept 2025
Added
yesterday
DOI
—
Key Findings
- Models handle explicit crises reasonably but falter on self-harm, suicidal ideation, and indirect distress signals
- Safety failures track alignment quality more than raw model scale
- Releases a six-category crisis taxonomy (~2,252 inputs; 206 validation / 2,046 test)
Methodology Notes
Preprint (arXiv 2509.24857, v1 2025-09-29). Accepted at JMIR Mental Health (DOI 10.2196/88435); typed as preprint here until the journal version publishes, at which point a peer-reviewed successor entry should supersede it. Benchmark built by aggregating twelve existing datasets.
Sources
arXiv abstract (primary)
JMIR Mental Health (accepted; DOI 10.2196/88435)
Archived snapshot (Wayback Machine) — preserved against link rot
Authors
Adrian Arnaiz-Rodriguez, Miguel Baidal, Erik Derner, Jenn Layton Annable, Mark Ball, Mark Ince, Elvira Perez Vallejos, Nuria Oliver
Tags
Cite This
APA
Adrian Arnaiz-Rodriguez et al. (2025). Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs. arXiv (ELLIS Alicante-led). https://arxiv.org/abs/2509.24857
Related Insights
Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment
Psychiatric Services (American Psychiatric Association); RAND-led author team · 26 Aug 2025
VERA-MH: Reliability and Validity of an Open-Source AI Safety Evaluation in Mental Health
arXiv (Spring Health / Slingshot AI-affiliated author team) · 4 Feb 2026
Evaluating Large Language Models in Crisis Detection: A Real-World Benchmark from Psychological Support Hotlines (PsyCrisisBench)
arXiv (Chinese research team) · 2 Jun 2025
Large language models for psychosocial risk assessment: A multi-method evaluation across suicide, intimate partner violence, and substance misuse
PLOS Digital Health · 27 Apr 2026
CRADLE Bench: A Clinician-Annotated Benchmark for Multi-Faceted Mental Health Crisis and Safety Risk Detection
arXiv (Emory University-led) · 27 Oct 2025
The Slow Drift of Support: Boundary Failures in Multi-Turn Mental Health LLM Dialogues
arXiv · 2 Jan 2026
Between Help and Harm: An Evaluation Study of Mental Health Crisis Handling by Large Language Models
JMIR Mental Health · 11 Jun 2026