Benchmark / dataset Credible

Evaluating Large Language Models in Crisis Detection: A Real-World Benchmark from Psychological Support Hotlines (PsyCrisisBench)

PsyCrisisBench is a real-world crisis-detection benchmark built from 540 annotated transcripts from a psychological support hotline in Hangzhou, China. It evaluates 64 models on mood recognition, suicidal-ideation detection, plan identification, and risk evaluation.

Read at arxiv.org Archived copy

Publisher

arXiv (Chinese research team)

Published

2 Jun 2025

Added

yesterday

DOI

10.48550/arXiv.2506.01329

Key Findings

F1 up to 0.88-0.91 on suicide-related tasks, with mood recognition the hardest (F1 ~0.709)
A fine-tuned small model outperformed larger general models on several tasks
Provides a non-English, real-world crisis-detection benchmark grounded in hotline transcripts

Methodology Notes

Preprint (arXiv 2506.01329, v1 2025-06-02; v2 2025-12-17). 540 real hotline transcripts (Chinese) with expert annotation across four crisis tasks; real-world rather than synthetic data.

Sources

arXiv abstract (primary)

Archived snapshot (Wayback Machine) — preserved against link rot

Topics

Crisis detection Suicide risk assessment Benchmarks Eval methodology Clinical integration

Authors

Guifeng Deng, Shuyin Rao, Tianyu Lin

Cite This

APA

Guifeng Deng, Shuyin Rao, Tianyu Lin (2025). Evaluating Large Language Models in Crisis Detection: A Real-World Benchmark from Psychological Support Hotlines (PsyCrisisBench). arXiv (Chinese research team). https://arxiv.org/abs/2506.01329

Related Insights

Preprint

Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs

arXiv (ELLIS Alicante-led) · 29 Sept 2025

Benchmark / dataset

VERA-MH: Reliability and Validity of an Open-Source AI Safety Evaluation in Mental Health

arXiv (Spring Health / Slingshot AI-affiliated author team) · 4 Feb 2026

Peer-reviewed

Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment

Psychiatric Services (American Psychiatric Association); RAND-led author team · 26 Aug 2025

Peer-reviewed

A machine learning approach to identifying suicide risk among text-based crisis counseling encounters

Frontiers in Psychiatry · 23 Mar 2023