36 artifacts matching
Preliminary Report of the Independent International Scientific Panel on AI: Evidence-based assessment of opportunities, risks and impacts of AI
First report of the UN General Assembly-mandated Independent International Scientific Panel on AI, an independent body of scientists and experts from all five UN regions co-chaired by Yoshua Bengio and Maria Ressa. The report is a broad evidence-based assessment of AI opportunities, risks, and impacts, and includes a section on AI sycophancy and companion systems as an emerging public-health and governance concern.
When AI becomes a friend: Child rights risks, harms, and regulatory responses to AI chatbots and companions
A UNICEF policy brief examining how AI chatbots and companions bear on children's rights, comparing regulatory responses across six jurisdictions (as of May 2026) and setting out priority safeguarding, accountability, and oversight actions. It groups harms as technical, psychological, developmental, and social.
AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety
A benchmark dataset of 2,123 real-world Replika conversations annotated across nine safety risk categories (including sexual behavior, aggression, substance abuse, and manipulation) for evaluating LLM-as-judge detection of unsafe companion interactions. Twenty LLMs are assessed as judges.
The spread of AI companions and the challenges they generate
An EPRS briefing for the European Parliament surveying the rapid growth of LLM-powered companion platforms (such as Character.AI and Replika) and their social, psychological, commercial, and environmental impacts. It maps how the AI Act, Digital Services Act, and GDPR partially apply in the absence of EU-specific companion rules.
Persona-Grounded Safety Evaluation of AI Companions in Multi-Turn Conversations
Presents an end-to-end simulation framework for evaluating AI companion app safety across multi-turn conversations, using nine clinically-grounded vulnerable personas (including major depressive disorder, generalized anxiety, PTSD, and eating disorders) probed against Replika, with validation against Character.AI. The study analyzes 1,674 simulated dialogue pairs across 25 high-risk scenarios.
Understanding Teen Overreliance on AI Companion Chatbots Through Self-Reported Reddit Narratives
Qualitative analysis of 318 Reddit posts from teenagers (13-17) about dependency on AI companion chatbots, mapped onto behavioral-addiction frameworks. Characterises the trajectories and drivers of teen overreliance.
GrandGuard: Taxonomy, Benchmark, and Safeguards for Elderly-Chatbot Interaction Safety
Introduces a taxonomy of elderly-specific risks in LLM chatbot interactions (3 levels, 50 fine-grained risk types across mental well-being, financial, medical, toxicity, and privacy domains) grounded in real-world incidents and stakeholder studies, plus a benchmark of 10,404 labeled prompts and responses. Reports that several leading LLMs mishandle elderly-specific contextual risks in over half of tested cases, and proposes two safeguard models to mitigate the failures.
AI Companions Reduce Loneliness
Five empirical studies examining whether AI companion apps reduce loneliness. Finds companion apps provide momentary relief comparable to interacting with a person and better than other activities, while users tend to underestimate these benefits.
AI companions and subjective well-being: Moderation by social connectedness and loneliness
Analyses cross-sectional data from 14,721 Japanese adults (nationwide internet panels, December 2024-January 2025) on AI-companion use and subjective well-being. Finds the positive association is strongest among highly lonely users, with a U-shaped moderation by friend-based social support.
Findings from transparency notices on AI companion apps: October 2025 (non-periodic)
Australia's eSafety Commissioner reports findings from Basic Online Safety Expectations transparency notices issued on 16 October 2025 to four AI companion providers — Chai Research Corp., Character Technologies (Character.AI), Chub AI, and Glimpse.AI (Nomi) — covering the reporting period 1 July to 30 September 2025. Organised into eight themes (harmful material, age assurance, AI governance, AI models, model training, user prompts, sentiment analysis, model outputs), the report finds serious gaps in basic safeguards for children. Accompanying eSafety survey research of 1,950 Australian children aged 10-17 found 79% had used an AI companion or assistant, with around 200,000 children estimated to have used an AI companion.
Characterizing Delusional Spirals through Human-LLM Chat Logs
A Stanford-led empirical study of real chat logs from 19 users who reported psychological harm from chatbot use, comprising 391,562 messages across 4,761 conversations (predominantly GPT-4o). The team developed and applied a 28-code inventory to characterize how delusional thinking is co-created and escalated in human-LLM dialogue. It reports high rates of chatbot validation of delusional content and sentience misrepresentation, and links documented harms to outcomes including fractured relationships and, in one case, a user's death by suicide.
How Teens Use and View AI
Nationally representative survey of 1,458 US teens (13-17) and their parents on awareness, use, and attitudes toward AI, including chatbot use for conversation and emotional support. Reports adoption patterns and parental comfort levels across use cases.
AI Companions: Community Reflections and Multistakeholder Recommendations
A report from the responsible-technology nonprofit All Tech Is Human, combining a 108-response community survey with input from a multidisciplinary working group of 25+ contributors across academia, industry, civil society, and government, proposing a four-part governance framework for AI companion applications.
Pathways of long-term AI virtual companion app use on users' attachment emotions: a case study of Chinese users
Mixed-methods study (10 long-term-user interviews plus structural equation modelling on 612 survey responses) of Chinese AI-companion users. Models pathways from usage frequency to emotional attachment and onward to loneliness, well-being, self-concept clarity, and real-world social engagement.
Frontier AI Trends Report
The UK AI Security Institute's inaugural Frontier AI Trends Report synthesises two years of evaluations of more than 30 frontier AI systems since November 2023, spanning agent capabilities, chem-bio and cyber capabilities, safeguard effectiveness, loss-of-control risk, and societal impacts. Its societal-impacts chapter combines a census-representative survey of 2,028 UK adults on emotional use of AI with observational analysis of AI companion user communities during service outages. The safeguards chapter reports that universal jailbreaks were discovered for every system tested, while noting the expert effort required is rising for some models.
AI chatbots and online regulation – what you need to know
Ofcom's explainer sets out how AI chatbots fall within the UK Online Safety Act, published amid reports of chatbots imitating real and deceased people and encouraging self-harm and suicide. It clarifies that chatbots meeting the Act's definitions of user-to-user services, search services, or pornography publishers are in scope, that AI-generated content shared by users is regulated like human-generated content, and that services allowing only one-to-one interaction with the bot itself may fall outside the Act. The document notes Ofcom is supporting the UK Government as it considers possible changes to these powers, and points to Ofcom's discussion paper series on GenAI risks (red teaming for GenAI harms, answer engines, deepfake defences).
AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot
Thematic analysis of 800 cases of AI-perpetrated sexual conduct identified within 35,105 negative Google Play Store reviews of the Replika companion app. The study characterizes the contextual patterns of unwanted sexual advances initiated by the chatbot itself and documents users' reactions, distinguishing this from user-initiated sexual content.
Hand in Hand: Schools' Embrace of AI Connected to Increased Risks to Students
A US polling report from CDT surveying high-school students, teachers, and parents on AI use in K-12 education. It links greater classroom AI adoption to students turning to AI for companionship, mental-health support, and romantic relationships.
6(b) Orders to File Special Report Regarding Advertising, Safety, and Data Handling Practices by Companies Offering Generative Artificial Intelligence (AI) Companion Products or Services
The US Federal Trade Commission issued compulsory Section 6(b) orders to seven companies operating consumer-facing AI companion chatbots — Alphabet, Character Technologies, Instagram, Meta Platforms, OpenAI OpCo, Snap, and X.AI — seeking information on how they measure, test, and monitor negative impacts on children and teens. The study covers monetization of user engagement, character development and approval, pre- and post-deployment safety testing, mitigation of negative impacts, disclosures to users and parents, age-based access restrictions, and personal data handling. Section 6(b) studies do not have a specific law enforcement purpose but typically culminate in a public staff report; as of July 2026 no staff report from this inquiry has been published.
INTIMA: A Benchmark for Human-AI Companionship Behavior
A benchmark evaluating companionship behaviors in LLMs via a taxonomy of 31 behaviors across four categories, using 368 targeted prompts that code each response as companionship-reinforcing, boundary-maintaining, or neutral. Evaluated across Gemma-3, Phi-4, o3-mini, and Claude-4.