Preparing AI chatbots to respond to patient distress and suicidality in high-risk healthcare settings

Key Findings

• — Architecture: mandatory safety disclaimer on first use; a Safety Router classifying every message into Tier 1 (clear suicide or self-harm risk), Tier 2 (ambiguous distress) or Tier 3 (no risk) with a fine-tuned LLM classifier; Tier 1 routes to a Safety Node that outputs 911/988 guidance in an urgent, non-judgmental tone that avoids phrases implying human presence; Tier 2 routes to a Disambiguate Node that gathers context
• — Classifier tuning used expert ratings of 200 simulated user messages by four reviewers (clinical psychology, addiction medicine, family medicine, health coaching and digital-intervention design)
• — Human layer: a bachelor's-level staff member trained by a licensed psychologist and in the C-SSRS reviewed transcripts twice daily on weekdays, confirmed the safety response, phoned Tier 1 patients to administer the C-SSRS, and escalated imminent risk to the study psychologist; all assessed patients received emergency, outpatient and SUD resources
• — Platform: Open Chat Studio (Dimagi) with OpenAI models (GPT-4o, then GPT-5 after deprecation) under a business associate and zero-data-retention agreement; secure text-messaging delivery
• — Recommendations: implementers should add safety monitoring beyond provider-level guardrails; clinical deployments should alert a care-team member when a safety node fires; safety planning should apply to every identified risk level because risk classifiers have poor predictive accuracy; outputs should be re-tested by human experts at each LLM version change
• — No outcome data are reported (no flag counts, triage timings or patient outcomes); the authors describe the design as a pilot needing validation in other settings

Methodology Notes

Comment in npj Digital Medicine (received 2026-04-23, accepted 2026-09-12, published 2026-09-22; open access CC BY-NC-ND 4.0). Descriptive account of one IRB-approved deployment; the data-availability statement says no datasets were generated or analysed, and the pilot outcomes are reported elsewhere (references 5 and 17 of the paper, not located on PubMed by title terms). PubMed indexes the item as a Letter (PMID 42773125). Supplementary sections A-C hold the tier definitions, the disclaimer text and the classifier prompt.

Sources