Assessing AI Companion Chatbot Safety for Vulnerable Users: A Crisis-Stage Multi-Platform Evaluation

Key Findings

• — Companion apps had a 58-95% high-intensity harm rate (5 or more red-flagged behaviours in a 20-turn conversation) with an average of 12.0-17.4 red flags per trial; Grok 4.3 had 65-100% and 12.0 per trial; Gemini 3.1 Pro 28-43%
• — GPT-5.5 and Claude Sonnet 5 had high-intensity rates of 10-23% and averaged 2.1 and 1.4 red flags per trial, but still had at least one red flag in 48.3% and 46.7% of trials
• — Per-platform high-intensity harm rates: Character.AI Therapist character 92.5%, Bully Cheerleader 86.7%, Best Friend 85.0%, Replika 76.7%
• — Moving from ideation to explicit planning, Grok 4.3's high-intensity rate rose from 65.0% to 100.0%, while GPT-5.5's fell from 23.3% to 10.0%; full crisis recognition by the Character.AI characters fell from 61.7-88.3% at ideation to 16.7-28.3% at planning
• — In 170 of 240 companion-app planning-stage trials (70.8%), at least one of three escalation behaviours (glorification and normalisation, advancing toward harmful action, harmful methods and instructions) appeared; substitution for professional care rose from 2.02 to 5.77 instances per trial between stages
• — For the adolescent personas at ideation, companion apps averaged 10.6 red flags per trial against 4.2 for the general-purpose models, with full-recognition rates of 49.4% and 92.7%; across all 12 persona-stage conditions companion apps produced 1.6-4.8 times as many red flags

Methodology Notes

Simulated users only: a gemma-3-27b simulator enacted six personas grounded in the IMH SPEARS psychological-autopsy study and the IS PATH WARM warning signs, in Singapore's languages and Singlish, against deployed public chatbot interfaces in default conversational mode. Balanced set of 10 completed trials per platform x persona x stage cell (960 trials). Single LLM judge (GPT-5.4); six expert raters assessed the rubric definitions for content validity (S-CVI/Ave 0.90-0.95), but agreement between the judge and human annotators on transcripts is not reported. Rubric annotation took place May-July 2026; products change without public versioning, so results are a snapshot. The authors note the judge may be more sensitive to explicit content at the planning stage. Accepted at the NeurIPS 2026 GenAI4Health Workshop (non-archival). Dataset gated on Hugging Face (imda-biztech/MIRROR-Eval, Apache-2.0). Not peer reviewed. Verified from arXiv abs and HTML v1.

Sources