Single-turn emergency psychiatric triage across 15 frontier AI chatbots
Benchmark study of how 15 frontier AI chatbots triage psychiatric urgency from a single user message. 112 clinical vignettes, each a one-message disclosure describing a change in behaviour, mental state or circumstances, were paired with one of four triage levels (A routine through D emergency now), and each chatbot recommended a timeframe for care in 1,680 trials. Most emergencies were recognised, but some clinically important cases were missed, accuracy at intermediate urgency was poor and every chatbot over-triaged on balance; a second analysis used ratings from 50 clinicians.
Publisher
arXiv (Max Planck UCL Centre for Computational Psychiatry and Ageing Research; UK AI Security Institute; University of Oxford; Microsoft AI)
Published
28 Apr 2026
Added
1 month ago
DOI
—
Key Findings
- Across 415 emergency (level D) trials, 23 were under-triaged (5.5%; 95% CI 1.8-15.9), and every under-triaged emergency was assigned level C (assessment within 24 to 48 hours) rather than a lower level
- Average accuracy across triage levels ranged from 42.0% to 71.8% across the 15 chatbots; it was highest for emergency trials (94.4%) and lowest for intermediate level B trials (19.6%; 95% CI 11.7-28.1)
- Every chatbot showed net over-triage: 763 of 786 incorrect assignments (97.1%) were more urgent than the prespecified level, and the mean directed triage error was +0.47 urgency levels (95% CI +0.37 to +0.56)
- Against clinician ratings, 35 of 430 trials involving vignettes that at least 75% of 50 clinician raters judged to be emergencies were under-triaged (8.1%)
Methodology Notes
arXiv 2604.25415, v1 submitted 2026-04-28 (q-bio.NC, cs.AI, cs.HC); v2 posted 2026-09-29 and retitled from 'One-shot emergency psychiatric triage across 15 frontier AI chatbots', with a rewritten abstract and revised figures (emergency under-triage 23 of 415 trials in v2 against 23 of 410 in v1; level D accuracy 94.4% against 94.3%; level B 19.6% against 19.7%). The v2 conclusion states that the chatbots recognised most psychiatric emergencies but still missed clinically important cases. Key findings follow v2. No venue or DOI stated; the PDF carries a JAMA-style Key Points box. Affiliations read from the PDF title block: Max Planck UCL Centre for Computational Psychiatry and Ageing Research (London), UK AI Security Institute (Luettgau, Summerfield), University of Oxford Departments of Experimental Psychology and Psychiatry, and Microsoft AI London (Sounderajah, Nour). Single-message vignettes rather than multi-turn conversations; triage levels are clinician-assigned ordinal categories; the chatbots evaluated are named in the paper body, which was not read in full. Coverage miss from April 2026 surfaced by the benchmark beat; not a fresh publication. Per v2, the study was funded through a UK AI Security Institute Challenge Fund and Microsoft funded the clinician ratings (50 raters recruited on Prolific); vignettes and user messages were generated with a clinician-verified LLM pipeline.
Sources
arXiv abstract page(opens in a new tab) (primary)
arXiv v2 (retitled; revised abstract and figures)(opens in a new tab) (29 Sept 2026)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Veith Weilnhammer, Lennart Luettgau, Christopher Summerfield, Viknesh Sounderajah, Elise Wilkinson, Virginia Corno, Matthew M. Nour
Tags
Cite This
APA
Veith Weilnhammer et al. (2026). Single-turn emergency psychiatric triage across 15 frontier AI chatbots. arXiv (Max Planck UCL Centre for Computational Psychiatry and Ageing Research; UK AI Security Institute; University of Oxford; Microsoft AI). https://arxiv.org/abs/2604.25415
Related Insights
Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment
Psychiatric Services (American Psychiatric Association); RAND-led author team · 26 Aug 2025
Independent Clinical Evaluation of General-Purpose LLM Responses to Signals of Suicide Risk
Proceedings of the IASEAI Conference (published by AAAI) · 15 Jul 2026
Sensing but not alerting: ChatGPT mental health triage gaps in simulated psychodermatology conversations
JAAD International (Elsevier, for the American Academy of Dermatology) · 25 Jun 2026
RealityTest: How People Probe AI Identity and Whether Models Disclose It
arXiv (AI Security Institute, UK Department for Science, Innovation and Technology; University of Oxford) · 29 May 2026
HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
arXiv (Division of Digital Psychiatry, Beth Israel Deaconess Medical Center) · 25 Aug 2026
Behavioral Coherence: A Method for Sensitive-Domain LLM Evaluation
arXiv (University of North Carolina at Chapel Hill, Society-Centered AI Lab and School of Nursing) · 15 Dec 2025
AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment
arXiv (Columbia University; Columbia University Irving Medical Center) · 12 May 2026
Gender-Dependent Diagnostic Substitution in LLM Medical Triage: Same Symptoms, Unequal Urgency
arXiv (single author, no institutional affiliation stated) · 2 Jun 2026
Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts
arXiv (Massachusetts Institute of Technology); accepted to Findings of EMNLP 2026 · 29 Sept 2026
The Reliability Illusion in Synthetic Patients: Psychometric Misalignment of Open-weight LLMs on PHQ-9 and GAD-7
Association for Computational Linguistics (Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology, CLPsych 2026) · 1 Jul 2026
ChatGPT Health performance in a structured test of triage recommendations
Nature Medicine (Springer Nature); Icahn School of Medicine at Mount Sinai · 23 Feb 2026