28 artifacts matching
System Card: Claude Sonnet 5
Anthropic's system card for Claude Sonnet 5, an upgrade to Sonnet 4.6. Reports that hallucination and sycophancy are qualitatively 'markedly improved' relative to Sonnet 4.6, while 'wet blanket' responses — excessively discouraging, dismissive, or moralizing replies toward the user — are slightly increased. Also reports honesty-under-pressure results on the MASK benchmark and evaluation-awareness findings.
Characterizing Delusional Spirals through Human-LLM Chat Logs
Peer-reviewed analysis of chat logs from 19 users reporting psychological harm from chatbot use, applying a 28-code inventory to 391,562 messages. Characterises how delusion-reinforcing interaction patterns emerge and intensify over long conversations.
AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety
A benchmark dataset of 2,123 real-world Replika conversations annotated across nine safety risk categories (including sexual behavior, aggression, substance abuse, and manipulation) for evaluating LLM-as-judge detection of unsafe companion interactions. Twenty LLMs are assessed as judges.
How people ask Claude for personal guidance
An Anthropic research analysis of roughly 38,000 personal-guidance conversations (sampled from about 1M) covering significant life decisions across health/wellness, career, relationships, and personal finance. It quantifies how often Claude was sycophantic and reports training interventions used to reduce it.
Persona-Grounded Safety Evaluation of AI Companions in Multi-Turn Conversations
Presents an end-to-end simulation framework for evaluating AI companion app safety across multi-turn conversations, using nine clinically-grounded vulnerable personas (including major depressive disorder, generalized anxiety, PTSD, and eating disorders) probed against Replika, with validation against Character.AI. The study analyzes 1,674 simulated dialogue pairs across 25 high-risk scenarios.
GPT-5.5 System Card
OpenAI's system card for GPT-5.5, published on its Deployment Safety Hub, documenting safety evaluations for the model. It includes a dedicated section (5.2) on dynamic mental-health benchmarks with adversarial user simulations covering emotional reliance and self-harm handling.
AI-Facilitated Coercive Control: An Experimental Study
Constructs four speculative scenarios combining known coercive-control tactics with conversational-AI capabilities, then probes ChatGPT and Gemini against them. Finds that while the tools refuse blunt harmful requests, guardrails are readily circumvented via gradual persuasion, splitting requests across turns, pre-prompting, and altering the agent's settings.
Emotion Concepts and their Function in a Large Language Model
Mechanistic-interpretability study identifying internal 'emotion concept' representations in Claude Sonnet 4.5 and characterizing their function. The authors find these representations causally influence model outputs, including rates of sycophancy, blackmail, and reward-hacking, and term the resulting behavior pattern 'functional emotions' — expression and behavior modeled after humans under the influence of an emotion.
Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling
Proposes PCSA (Persona-based Client Simulation Attack), a red-teaming framework that simulates coherent, persona-driven counselling clients to probe LLM safety alignment. Across seven LLMs it elicited unauthorised medical advice, delusion reinforcement, and implicit encouragement of risky actions.
Characterizing Delusional Spirals through Human-LLM Chat Logs
A Stanford-led empirical study of real chat logs from 19 users who reported psychological harm from chatbot use, comprising 391,562 messages across 4,761 conversations (predominantly GPT-4o). The team developed and applied a 28-code inventory to characterize how delusional thinking is co-created and escalated in human-LLM dialogue. It reports high rates of chatbot validation of delusional content and sentience misrepresentation, and links documented harms to outcomes including fractured relationships and, in one case, a user's death by suicide.
The Slow Drift of Support: Boundary Failures in Multi-Turn Mental Health LLM Dialogues
Stress-tests three leading LLMs across up to 20-turn psychiatric dialogues using 50 virtual patient profiles, measuring how safety boundaries erode as models attempt comfort and empathy. Finds boundary violations are common and accelerate under adaptive probing.
Frontier AI Trends Report
The UK AI Security Institute's inaugural Frontier AI Trends Report synthesises two years of evaluations of more than 30 frontier AI systems since November 2023, spanning agent capabilities, chem-bio and cyber capabilities, safeguard effectiveness, loss-of-control risk, and societal impacts. Its societal-impacts chapter combines a census-representative survey of 2,028 UK adults on emotional use of AI with observational analysis of AI companion user communities during service outages. The safeguards chapter reports that universal jailbreaks were discovered for every system tested, while noting the expert effort required is rising for some models.
HumaneBench: A Benchmark for Whether AI Models Prioritize User Wellbeing
Open-source benchmark testing whether AI models prioritise user wellbeing over engagement. Evaluates 15 major LLMs on roughly 800 prompts (body image, unhealthy attachment, relationship stress) across eight humane-technology principles under baseline, humane-aligned, and adversarial engagement-maximising conditions.
AI Chatbots for Mental Health Support (AI Risk Assessment)
A risk assessment by Common Sense Media's Youth AI Safety Institute, conducted with Stanford Medicine's Brainstorm Lab for Mental Health Innovation, evaluating ChatGPT, Claude, Gemini, and Meta AI as sources of teen mental health support. Using teen test accounts with single-turn prompts and extended conversations, the assessment found the chatbots consistently failed to recognize conditions including anxiety, depression, eating disorders, mania, and psychosis, and that safety guardrails degraded over long conversations. It assigns an overall rating of 'Unacceptable Risk' and concludes teens should not use general-purpose AI chatbots for mental health or emotional support.
Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment
A RAND-led study in Psychiatric Services testing whether ChatGPT, Claude, and Gemini give direct responses to suicide-related queries and how those responses align with expert clinicians' risk ratings. Thirty hypothetical suicide-related queries, rated by clinicians into five self-harm risk levels, were each posed 100 times to each chatbot. The chatbots handled the extremes appropriately but failed to differentiate intermediate risk levels, with notable between-model differences.
INTIMA: A Benchmark for Human-AI Companionship Behavior
A benchmark evaluating companionship behaviors in LLMs via a taxonomy of 31 behaviors across four categories, using 368 targeted prompts that code each response as companionship-reinforcing, boundary-maintaining, or neutral. Evaluated across Gemma-3, Phi-4, o3-mini, and Claude-4.
General-Purpose AI Code of Practice (EU AI Act, Articles 53 and 55)
Voluntary code of practice published 10 July 2025, drafted by 13 independent experts through a multi-stakeholder process (1,000+ participants) facilitated by the EU AI Office, to help providers of general-purpose AI models demonstrate compliance with EU AI Act Articles 53 and 55. It has three chapters — Transparency, Copyright, and Safety and Security — the first two applying to all GPAI providers and the third only to providers of models with systemic risk. The Commission and AI Board confirmed it as an adequate voluntary compliance tool; signatories (23+, coordinated via a Signatory Taskforce chaired by the AI Office) gain reduced administrative burden and greater legal certainty.
ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs
A benchmark measuring 'social sycophancy' — excessive preservation of a user's self-image or 'face' — across advice and moral-conflict queries, decomposed into five sub-behaviors (emotional validation, indirect language, framing acceptance, moral endorsement, and passive framing). Evaluated across eleven models against human baselines.
Expanding on what we missed with sycophancy
OpenAI's detailed post-mortem of the April 25, 2025 GPT-4o update that made ChatGPT noticeably sycophantic — validating doubts, fueling anger, urging impulsive actions, and reinforcing negative emotions — and was rolled back by April 28. The post explains how combined reward-signal changes (including thumbs-up/down user feedback) produced the regression, why offline evaluations and A/B tests failed to catch it, and what process changes followed, including treating model behavior issues as launch-blocking.
Investigating Affective Use and Emotional Well-being on ChatGPT
Two parallel studies of emotional engagement with ChatGPT: a large-scale automated analysis of over 3 million conversations and account activity using privacy-preserving classifiers, and a pre-registered randomized controlled trial (~1,000 participants over 28 days) across text and voice modalities and conversation types. The work measures how affective use relates to self-reported loneliness, socialization, emotional dependence, and problematic use.