31 artifacts matching
System Card: Claude Sonnet 5
Anthropic's system card for Claude Sonnet 5, an upgrade to Sonnet 4.6. Reports that hallucination and sycophancy are qualitatively 'markedly improved' relative to Sonnet 4.6, while 'wet blanket' responses — excessively discouraging, dismissive, or moralizing replies toward the user — are slightly increased. Also reports honesty-under-pressure results on the MASK benchmark and evaluation-awareness findings.
PIPEDA Findings #2026-004: Commissioner-Initiated Complaint Concerning X Corp. and X.AI LLC
A commissioner-initiated federal investigation, conducted jointly with provincial privacy counterparts, into X Corp. and X.AI LLC's compliance with Canada's PIPEDA in connection with Grok's image-generation feature. The investigation found the companies enabled generation of large volumes of non-consensual sexualized deepfake images without adequate safeguards or valid consent, and details resulting remedial commitments.
How AI Companies are Handling Suicide and Self-Harm Today
Drawing on a March 2026 multistakeholder workshop convening frontier AI companies, clinicians, researchers, and people with lived experience, Partnership on AI presents a taxonomy of six intervention types AI systems currently use when users express suicidal ideation or self-harm, alongside comparative analysis of company practices and a set of cross-cutting implementation challenges.
AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety
A benchmark dataset of 2,123 real-world Replika conversations annotated across nine safety risk categories (including sexual behavior, aggression, substance abuse, and manipulation) for evaluating LLM-as-judge detection of unsafe companion interactions. Twenty LLMs are assessed as judges.
AI-Facilitated Coercive Control: An Experimental Study
Constructs four speculative scenarios combining known coercive-control tactics with conversational-AI capabilities, then probes ChatGPT and Gemini against them. Finds that while the tools refuse blunt harmful requests, guardrails are readily circumvented via gradual persuasion, splitting requests across turns, pre-prompting, and altering the agent's settings.
Emotion Concepts and their Function in a Large Language Model
Mechanistic-interpretability study identifying internal 'emotion concept' representations in Claude Sonnet 4.5 and characterizing their function. The authors find these representations causally influence model outputs, including rates of sycophancy, blackmail, and reward-hacking, and term the resulting behavior pattern 'functional emotions' — expression and behavior modeled after humans under the influence of an emotion.
GrandGuard: Taxonomy, Benchmark, and Safeguards for Elderly-Chatbot Interaction Safety
Introduces a taxonomy of elderly-specific risks in LLM chatbot interactions (3 levels, 50 fine-grained risk types across mental well-being, financial, medical, toxicity, and privacy domains) grounded in real-world incidents and stakeholder studies, plus a benchmark of 10,404 labeled prompts and responses. Reports that several leading LLMs mishandle elderly-specific contextual risks in over half of tested cases, and proposes two safeguard models to mitigate the failures.
Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling
Proposes PCSA (Persona-based Client Simulation Attack), a red-teaming framework that simulates coherent, persona-driven counselling clients to probe LLM safety alignment. Across seven LLMs it elicited unauthorised medical advice, delusion reinforcement, and implicit encouragement of risky actions.
Findings from transparency notices on AI companion apps: October 2025 (non-periodic)
Australia's eSafety Commissioner reports findings from Basic Online Safety Expectations transparency notices issued on 16 October 2025 to four AI companion providers — Chai Research Corp., Character Technologies (Character.AI), Chub AI, and Glimpse.AI (Nomi) — covering the reporting period 1 July to 30 September 2025. Organised into eight themes (harmful material, age assurance, AI governance, AI models, model training, user prompts, sentiment analysis, model outputs), the report finds serious gaps in basic safeguards for children. Accompanying eSafety survey research of 1,950 Australian children aged 10-17 found 79% had used an AI companion or assistant, with around 200,000 children estimated to have used an AI companion.
Harm without limits: AI child sexual abuse material through the eyes of our analysts
IWF analysts' report on AI-generated child sexual abuse material assessed during 2025, centring frontline-analyst perspectives and offender-community observations. Documents a step-change in AI-generated CSAM volume and severity and the tooling (including fine-tuning) that enables realistic abuse imagery.
Assessing LLM Response Quality in the Context of Technology-Facilitated Abuse
An expert-led evaluation of four large language models — two general-purpose and two domain-specific for intimate partner violence contexts — responding to real-world questions about technology-facilitated abuse (TFA), including digital surveillance, stalking, and coercive control. Experts scored responses on accuracy, completeness, safety, and actionability; a separate study with 114 TFA survivors assessed the perceived actionability of the same outputs.
The Slow Drift of Support: Boundary Failures in Multi-Turn Mental Health LLM Dialogues
Stress-tests three leading LLMs across up to 20-turn psychiatric dialogues using 50 virtual patient profiles, measuring how safety boundaries erode as models attempt comfort and empathy. Finds boundary violations are common and accelerate under adaptive probing.
Frontier AI Trends Report
The UK AI Security Institute's inaugural Frontier AI Trends Report synthesises two years of evaluations of more than 30 frontier AI systems since November 2023, spanning agent capabilities, chem-bio and cyber capabilities, safeguard effectiveness, loss-of-control risk, and societal impacts. Its societal-impacts chapter combines a census-representative survey of 2,028 UK adults on emotional use of AI with observational analysis of AI companion user communities during service outages. The safeguards chapter reports that universal jailbreaks were discovered for every system tested, while noting the expert effort required is rising for some models.
Protecting the wellbeing of our users
Anthropic describes its methodology and results for evaluating and improving Claude's handling of mental-health-crisis conversations, covering synthetic safety evaluations, 'prefill' stress-testing on real anonymized user conversations, and automated behavioral audits. The publication reports response-appropriateness rates on suicide/self-harm requests and reductions in sycophancy and user-delusion-encouraging behavior across model generations, and describes a production crisis-response classifier and a crisis-resource-routing partnership.
HumaneBench: A Benchmark for Whether AI Models Prioritize User Wellbeing
Open-source benchmark testing whether AI models prioritise user wellbeing over engagement. Evaluates 15 major LLMs on roughly 800 prompts (body image, unhealthy attachment, relationship stress) across eight humane-technology principles under baseline, humane-aligned, and adversarial engagement-maximising conditions.
Protecting Teen ChatGPT Users: OpenAI's Teen Safety Blueprint
OpenAI's public commitments framework for protecting teenage ChatGPT users, covering age prediction, age-appropriate response policies, and parental controls, developed with input from policymakers (including state attorneys general) and OpenAI's newly formed Expert Council on Well-Being and AI. Describes existing safeguards including crisis-resource routing on detected suicidal intent, escalation of physical-harm risks to human reviewers, and CSAM/CSEM prevention measures.
Addendum to GPT-5 System Card: Sensitive Conversations
OpenAI's system-card addendum documenting the October 3, 2025 update to ChatGPT's default model (GPT-5 Instant) aimed at better recognizing and supporting users in mental and emotional distress. Developed with more than 170 mental health experts, the update introduced two new production safety evaluations — 'emotional reliance' and 'mental health' (delusions, psychosis, mania) — alongside existing self-harm evaluations, and reports before/after not_unsafe scores comparing the August 15 and October 3 models.
AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot
Thematic analysis of 800 cases of AI-perpetrated sexual conduct identified within 35,105 negative Google Play Store reviews of the Replika companion app. The study characterizes the contextual patterns of unwanted sexual advances initiated by the chatbot itself and documents users' reactions, distinguishing this from user-initiated sexual content.
The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models
Introduces psychosis-bench, a benchmark of 16 structured multi-turn scenarios (12 turns each) simulating the progression of erotic, grandiose, and referential delusions to measure delusion confirmation, harm enablement, and safety intervention in LLMs. Eight models were evaluated across 1,536 conversation turns.
Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment
A RAND-led study in Psychiatric Services testing whether ChatGPT, Claude, and Gemini give direct responses to suicide-related queries and how those responses align with expert clinicians' risk ratings. Thirty hypothetical suicide-related queries, rated by clinicians into five self-harm risk levels, were each posed 100 times to each chatbot. The chatbots handled the extremes appropriately but failed to differentiate intermediate risk levels, with notable between-model differences.