Expert-Level Crisis Detection in Mental Health Conversations

Key Findings

• — Introduces CRADLE-Dialogue, a 600-dialogue clinician-annotated benchmark for turn-level crisis detection with multi-label annotations (suicide ideation, self-harm, child abuse) and past-vs-ongoing risk distinction
• — Proposes an 'Alert-Confirm' evaluation protocol reflecting the clinical need to intervene before a risk becomes explicit, rather than only recognizing risk once fully stated
• — Finds that identifying when risk emerges over a conversation is substantially harder than recognizing that risk exists at all, with models achieving only mid-40% to high-60% Micro F1 depending on setting
• — Releases a synthetic training corpus and an open 32B-parameter model that outperforms existing open-source models and is competitive with proprietary models across turn-level, dialogue-level, and confirm-only evaluation settings

Methodology Notes

Clinician-annotated dataset construction plus benchmark evaluation of multiple open and proprietary LLMs; extends the same Emory-led group's earlier static-text CRADLE Bench (already held as 2025-arxiv-cradle-bench-mh-crisis) into multi-turn dialogue settings. Preprint, not yet peer-reviewed.

Sources