Before the Labels: How Dataset Construction Shapes Suicidality Detection in Clinical Text
A critical case study of the ScAN suicidality dataset built over MIMIC-III clinical notes, arguing that EHR-based suicidality datasets encode a particular operationalisation of suicidality shaped by who authors the notes, how episodes are bounded and how ambiguity is resolved. Governance constraints, ICD-based cohort selection, single-annotator labelling and hospital-stay-level aggregation produce labels that foreground clinician judgement and treat suicidality as a bounded episode; a linguistic analysis shows identical labels covering heterogeneous framings that differ in temporality, negation and uncertainty, and labelling patterns that differ by insurance status.
Publisher
Association for Computational Linguistics (Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology, CLPsych 2026); University of Washington
Published
1 Jul 2026
Added
today
Key Findings
- Identical suicidality labels in ScAN subsume heterogeneous clinical framings that differ in temporality, negation and uncertainty; the beat's PDF read records 27.8% of present-ideation spans carrying historical markers.
- Cohort selection by ICD code, single-annotator labelling with limited physician review, and aggregation to the hospital stay all shape what counts as suicidality before any model is trained.
- Labelling patterns differ across insurance status, and the underlying cohort skews to White, English-speaking patients, which the authors flag as a source of downstream disparity.
- The authors call on the clinical NLP community to examine the assumptions embedded in suicidality datasets before treating their labels as ground truth.
Methodology Notes
Critical dataset analysis with linguistic examination of annotated spans in ScAN (12,759 MIMIC-III notes, 669 patients, 19,690 span annotations per the beat's PDF read); no new patient data. Published in the CLPsych 2026 workshop proceedings (July 2026), pages 119 to 127, DOI 10.18653/v1/2026.clpsych-1.9. Verified at the ACL Anthology on 2026-09-15. The object is clinical text, not conversation; it enters the library as a methods caveat for suicide-risk detection systems.
Sources
Authors
Priyanshi Garg, Ishita Rao, Jieqiong Ding, Amandalynne Paullada
Tags
Cite This
APA
Priyanshi Garg et al. (2026). Before the Labels: How Dataset Construction Shapes Suicidality Detection in Clinical Text. Association for Computational Linguistics (Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology, CLPsych 2026); University of Washington. https://aclanthology.org/2026.clpsych-1.9/
Related Insights
Temporal and cross-site validation of an AI system for self-harm detection
PLOS Digital Health; RMIT University; University of Melbourne; Orygen · 11 Sept 2026
"Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated
arXiv (Washington University in St. Louis; Southern Methodist University) · 7 Aug 2025