When Chatbots Accommodate: Auditing the Response Policies of AI Companions in Vulnerable Conversations

Key Findings

• — Policies inferred from about 47,000 turns of real-world conversations on three named platforms (553 donated Replika and Character.AI transcripts plus 386 transcripts from a 110-participant four-week study with a GPT-4.1 chatbot), rather than from scripted crisis prompts
• — GPT-4.1 reaches for advice; Character.AI spreads its response across strategies with no dominant mode; Replika consistently asks questions and stays present
• — Over four weeks of repeated interaction, GPT-4.1 asks progressively fewer follow-up questions when users are distressed and, on internal distress, increasingly sets boundaries or refers users to outside help rather than pushing back
• — In exploratory comparisons, psychologically high-risk users (high PHQ-9, GAD-7 and loneliness scores) receive significantly fewer follow-up questions from GPT-4.1 when they express external distress
• — Replika users with a high companion bond receive significantly more advice on external distress; Character.AI high-bond users receive more emotional validation
• — The authors state that responses introducing corrective friction (follow-up questions and substantive pushback) recede over sustained interaction and that the estimated policies are invisible to output-level audits
• — Accepted to Findings of EMNLP 2026 (journal reference on the v2 abstract page)

Methodology Notes

Maximum Causal Entropy Inverse Reinforcement Learning applied to about 47,000 turns of real-world conversations across three platforms, plus a paired vulnerability-response taxonomy. The corpus comes from two earlier studies by the same group: 553 transcripts donated by real users (57 Replika, 496 Character.AI, spanning 112 user- or platform-created characters whose configurations are unobserved, so Character.AI estimates aggregate over heterogeneous personas) and 386 transcripts from a controlled study in which 110 participants interacted weekly over four weeks with a GPT-4.1 chatbot with persistent memory. The inferred policy is a model of observed behaviour, not access to any platform's actual objective, and the user-subgroup comparisons are described by the authors as exploratory. Preprint: v1 submitted 2026-06-03 under the title 'When Chatbots Accommodate: What AI Companions Optimize for in Vulnerable Conversations'; v2 posted 2026-08-31 with the current title and a journal reference to Findings of EMNLP 2026 (November 2026), cs.HC. The arXiv abstract renders the second platform name as 'this http URL' because arXiv mangles the domain-shaped string character.ai; the platform is Character.AI.

Sources