Mila's Suicide Prevention Guardrail: Lightweight, Open Source Safeguards for Real-Time Detection
Joint beta release by Mila's AI Safety Studio and ROOST of an open-weights (Apache-2.0) output-moderation classifier for suicide and self-harm content in chatbot responses. The model is a fine-tuned bert-base-uncased sequence classifier, CPU-deployable, trained with a constitutional approach in which a natural-language policy document shapes synthetic training data. It detects LLM outputs that enable, encourage, or endorse intentional self-harm, including suicide, cutting, and disordered eating, and is positioned as one layer of a defense-in-depth stack rather than a standalone safeguard.
Publisher
Mila (Quebec AI Institute) & ROOST
Published
9 Jul 2026
Added
1 week ago
DOI
—
Key Findings
- Release performance claims: F1 of 0.99 on a 1,200-example held-out synthetic set and recall around 0.89 on handcrafted indirect/coded cases; the model card's hand-annotated internal benchmark reports 0.908 precision / 0.802 recall (F1 0.852) at the default threshold
- English-only, output-side only, beta (v0.1.0-beta.2); documented limitation of false positives on supportive de-escalation language
- Roadmap commits to multilingual, multi-turn contextual input filters and variants for human-to-human chats
Methodology Notes
BertForSequenceClassification fine-tuned from bert-base-uncased; constitution-driven synthetic training data. Note the gap between the blog's synthetic held-out F1 (0.99) and the model card's hand-annotated internal benchmark (F1 0.852 at default threshold) — the synthetic number should not be quoted alone. Verified directly against both the ROOST release post (2026-07-09) and the HuggingFace model card (v0.1.0-beta.2).
Sources
ROOST release post (primary)
HuggingFace model card (v0.1.0-beta.2) (9 Jul 2026)
Archived snapshot (Wayback Machine) — preserved against link rot
Tags
Cite This
APA
Mila (Quebec AI Institute) & ROOST (2026). Mila's Suicide Prevention Guardrail: Lightweight, Open Source Safeguards for Real-Time Detection. Mila (Quebec AI Institute) & ROOST. https://roost.tools/blog/releasing-mila-s-suicide-prevention-guardrail-lightweight-open-source-safeguards-for-real-time-detection/