Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale

Key Findings

• — Six models were evaluated zero-shot across the C-SSRS 7-point severity scale, including Claude, GPT, Mistral and LLaMA variants
• — Claude and GPT aligned most closely with human C-SSRS annotations, while Mistral produced the lowest ordinal prediction error — so the best-agreeing model and the lowest-error model were not the same model
• — Most models showed ordinal sensitivity, with misclassifications concentrated between adjacent severity levels rather than scattered across the ladder
• — The authors conclude that human oversight, transparency and cautious deployment remain necessary, and publish code and supplementary materials

Methodology Notes

Conference proceedings version of arXiv:2505.13480 (submitted 11 May 2025), presented at IEEE FMLDS 2025 in Los Angeles, 2-5 November 2025, pages 245-254. Zero-shot classification against human-annotated Reddit posts; no fine-tuning, no clinical population, and severity labels derived from public forum text rather than clinical interview, so this measures instrument-shaped text classification and not C-SSRS administration. Author order differs between the preprint and the proceedings version. Verification route: IEEE Xplore (document 11446866) returns HTTP 202 bot-mitigation to this box, so the venue, page range, event dates, author list and 2025-11-02 publication date were confirmed from the Crossref DOI record for 10.1109/FMLDS67896.2025.00067; the substantive findings above were read from the openly available arXiv preprint of the same work. The proceedings full text was NOT read, so any change in results between the preprint and the version of record would not have been caught.

Sources