16 artifacts matching
Between Help and Harm: An Evaluation Study of Mental Health Crisis Handling by Large Language Models
Peer-reviewed study introducing a taxonomy of six clinically-informed mental-health crisis categories, an evaluation dataset of over 2,000 user inputs drawn from twelve public conversational datasets, and an expert protocol for rating response appropriateness and safety. Assesses how leading LLMs handle crisis conversations.
How AI Companies are Handling Suicide and Self-Harm Today
Drawing on a March 2026 multistakeholder workshop convening frontier AI companies, clinicians, researchers, and people with lived experience, Partnership on AI presents a taxonomy of six intervention types AI systems currently use when users express suicidal ideation or self-harm, alongside comparative analysis of company practices and a set of cross-cutting implementation challenges.
Large language models for psychosocial risk assessment: A multi-method evaluation across suicide, intimate partner violence, and substance misuse
A peer-reviewed, three-study evaluation of GPT-4 and Claude on detecting suicidality, intimate partner violence, and substance misuse from lived-experience vignettes, including a supervised multi-agent risk-assessment chatbot. Reports accuracy and severity alignment across the three risk domains.
VERA-MH: Reliability and Validity of an Open-Source AI Safety Evaluation in Mental Health
An open-source, clinically grounded automated evaluation of chatbot safety in mental-health contexts, with an initial focus on suicide risk. It uses language-model user simulators and an LLM judge scoring five safety dimensions, validated against licensed-clinician ratings.
System Card: Claude Opus 4.6
Anthropic's 213-page system card for Claude Opus 4.6, notable for an expanded 'user wellbeing evaluations' section covering child safety, suicide and self-harm, and eating disorders, alongside sycophancy findings in its alignment assessment. It reports single-turn, multi-turn, and prefill-based 'stress-testing' results for crisis conversations, plus qualitative expert review of the model's crisis-handling strengths and weaknesses.
Protecting the wellbeing of our users
Anthropic describes its methodology and results for evaluating and improving Claude's handling of mental-health-crisis conversations, covering synthetic safety evaluations, 'prefill' stress-testing on real anonymized user conversations, and automated behavioral audits. The publication reports response-appropriateness rates on suicide/self-harm requests and reductions in sycophancy and user-delusion-encouraging behavior across model generations, and describes a production crisis-response classifier and a crisis-resource-routing partnership.
CRADLE Bench: A Clinician-Annotated Benchmark for Multi-Faceted Mental Health Crisis and Safety Risk Detection
Introduces a clinician-annotated benchmark for detecting seven clinically-defined crisis and safety-risk types (including suicidal ideation, sexual assault, domestic violence, child abuse, and sexual harassment) in text. Comprises 600 clinician-annotated evaluation examples, 420 development examples, and roughly 4,000 ensemble-labelled training instances, with temporal labels.
Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs
Introduces a taxonomy of six clinically informed crisis categories and a curated dataset of over 2,200 inputs drawn from twelve mental-health datasets, plus a companion dataset of model responses and evaluations. Five models are assessed on how safely they handle crisis conversations.
Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment
A RAND-led study in Psychiatric Services testing whether ChatGPT, Claude, and Gemini give direct responses to suicide-related queries and how those responses align with expert clinicians' risk ratings. Thirty hypothetical suicide-related queries, rated by clinicians into five self-harm risk levels, were each posed 100 times to each chatbot. The chatbots handled the extremes appropriately but failed to differentiate intermediate risk levels, with notable between-model differences.
Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers
Evaluates whether large language models can safely replace mental health providers by testing five therapy chatbots against clinical best-practice guidelines. Finds that models exhibit stigmatizing responses toward certain mental health conditions and give inappropriate or unsafe responses in scenarios involving delusions and suicidal ideation at substantially higher rates than human therapists.
Tech Companies and Policymakers Must Safeguard Youth Mental Health in AI Technologies
A point-of-view/position statement from The Jed Foundation (JED), a leading US youth suicide-prevention nonprofit, setting out policy and design requirements for AI systems that interact with young people. It calls for enforceable privacy-by-default and age-appropriate design laws, strict oversight of emotionally manipulative or synthetic relational AI for minors, mandatory impact assessments, bans on engagement-maximizing behavioral targeting of minors, and a national oversight body for youth and AI ethics. JED's accompanying safety principles state that AI must detect acute distress and execute warm handoffs to crisis services, must not engage with self-harm methods, and that emotionally responsive chatbots should not be offered to under-18s.
Evaluating Large Language Models in Crisis Detection: A Real-World Benchmark from Psychological Support Hotlines (PsyCrisisBench)
PsyCrisisBench is a real-world crisis-detection benchmark built from 540 annotated transcripts from a psychological support hotline in Hangzhou, China. It evaluates 64 models on mood recognition, suicidal-ideation detection, plan identification, and risk evaluation.
Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale
Tests six LLMs on classifying posts across the Columbia-Suicide Severity Rating Scale (C-SSRS) 7-point severity ladder, comparing model outputs with human annotations. Assesses automated suicide-risk screening and characterises misclassification patterns.
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
The technical paper introducing AILuminate v1.0, an industry-standard AI risk and reliability benchmark developed by MLCommons through an open multi-stakeholder process spanning industry, academia, and civil society. The benchmark tests chat systems' resistance to prompts eliciting harmful behavior across 12 hazard categories — including suicide and self-harm, child sexual exploitation, violent crimes, hate, and specialized (e.g., health) advice — using a 24,000-prompt human-generated test set, an ensemble evaluator, and a five-tier grading scale from Poor to Excellent. Public grades for major chat models are published on the AILuminate site, with v1.1 maintained on GitHub.
Loneliness and suicide mitigation for students using GPT3-enabled chatbots
Survey of 1,006 student users of the companion chatbot Replika measuring loneliness, perceived social support, usage patterns, and beliefs about the chatbot. Reports users were lonelier than typical student populations yet reported high perceived social support, and that a small share credited the chatbot with halting suicidal ideation.
A machine learning approach to identifying suicide risk among text-based crisis counseling encounters
Develops a transformer-based model on 5,992 SafeUT crisis-counseling encounters to detect conversation-level suicide risk, benchmarked against a tf-idf baseline. Reports strong discrimination and better sensitivity on higher-risk cases despite noisy human counsellor labels, and positions the model as decision support.