AI-based detection of suicidal ideation in text: model development and evaluation for a student mental health chatbot

Key Findings

• — Encoder trained on 220,833 sentence-level samples from public social-media datasets; the selected configuration reached precision 0.91, recall 0.92, specificity 0.92, F1 0.91 and accuracy 0.92 on the held-out test set
• — A stricter cosine threshold (0.90 versus 0.81) raised precision to 0.977 and specificity to 0.993 but cut recall to 0.339 (false negatives rose from 851 to 6,699)
• — Performance was comparable to Phi-4 (accuracy 0.93, F1 0.92) at roughly one-eighteenth of the storage and under one-hundredth of the parameters
• — Proposed workflow: high-confidence detections trigger an immediate supportive response and crisis resources; intermediate scores prompt clarification and re-evaluation

Methodology Notes

Training and test data are public social-media datasets with source-assigned labels (the authors note label noise), not chatbot conversations; no deployment or user-outcome data. Version 2 published on OSF on 18 September 2026 (the OSF API does not expose a separate v1 record, so the row is dated to v2). Verified through the OSF API record and the deposited PDF read by the sweep.

Sources