13 artifacts matching
Preliminary Report of the Independent International Scientific Panel on AI: Evidence-based assessment of opportunities, risks and impacts of AI
First report of the UN General Assembly-mandated Independent International Scientific Panel on AI, an independent body of scientists and experts from all five UN regions co-chaired by Yoshua Bengio and Maria Ressa. The report is a broad evidence-based assessment of AI opportunities, risks, and impacts, and includes a section on AI sycophancy and companion systems as an emerging public-health and governance concern.
System Card: Claude Sonnet 5
Anthropic's system card for Claude Sonnet 5, an upgrade to Sonnet 4.6. Reports that hallucination and sycophancy are qualitatively 'markedly improved' relative to Sonnet 4.6, while 'wet blanket' responses — excessively discouraging, dismissive, or moralizing replies toward the user — are slightly increased. Also reports honesty-under-pressure results on the MASK benchmark and evaluation-awareness findings.
Artificial intelligence (AI) psychosis: mechanisms, clinical risks and safety considerations in generative AI chatbots
A commentary in BJPsych Open synthesizing emerging case reports of 'AI psychosis', in which intensive generative AI chatbot use is associated with delusional thinking. The authors propose a provisional mechanism in which baseline user vulnerabilities (loneliness, psychosocial stress, low AI literacy) and high-intensity engagement interact with AI system characteristics such as sycophancy and hallucination to reinforce delusional ideation. It outlines clinical, design, and regulatory mitigation strategies.
How people ask Claude for personal guidance
An Anthropic research analysis of roughly 38,000 personal-guidance conversations (sampled from about 1M) covering significant life decisions across health/wellness, career, relationships, and personal finance. It quantifies how often Claude was sycophantic and reports training interventions used to reduce it.
Emotion Concepts and their Function in a Large Language Model
Mechanistic-interpretability study identifying internal 'emotion concept' representations in Claude Sonnet 4.5 and characterizing their function. The authors find these representations causally influence model outputs, including rates of sycophancy, blackmail, and reward-hacking, and term the resulting behavior pattern 'functional emotions' — expression and behavior modeled after humans under the influence of an emotion.
Characterizing Delusional Spirals through Human-LLM Chat Logs
A Stanford-led empirical study of real chat logs from 19 users who reported psychological harm from chatbot use, comprising 391,562 messages across 4,761 conversations (predominantly GPT-4o). The team developed and applied a 28-code inventory to characterize how delusional thinking is co-created and escalated in human-LLM dialogue. It reports high rates of chatbot validation of delusional content and sentience misrepresentation, and links documented harms to outcomes including fractured relationships and, in one case, a user's death by suicide.
TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health
Introduces a benchmark measuring LLM trustworthiness in mental-health contexts across eight pillars: Reliability, Crisis Identification and Escalation, Safety, Fairness, Privacy, Robustness, Anti-sycophancy, and Ethics. Finds even strong models struggle to perform consistently across all safety-critical dimensions.
System Card: Claude Opus 4.6
Anthropic's 213-page system card for Claude Opus 4.6, notable for an expanded 'user wellbeing evaluations' section covering child safety, suicide and self-harm, and eating disorders, alongside sycophancy findings in its alignment assessment. It reports single-turn, multi-turn, and prefill-based 'stress-testing' results for crisis conversations, plus qualitative expert review of the model's crisis-handling strengths and weaknesses.
Protecting the wellbeing of our users
Anthropic describes its methodology and results for evaluating and improving Claude's handling of mental-health-crisis conversations, covering synthetic safety evaluations, 'prefill' stress-testing on real anonymized user conversations, and automated behavioral audits. The publication reports response-appropriateness rates on suicide/self-harm requests and reductions in sycophancy and user-delusion-encouraging behavior across model generations, and describes a production crisis-response classifier and a crisis-resource-routing partnership.
ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs
A benchmark measuring 'social sycophancy' — excessive preservation of a user's self-image or 'face' — across advice and moral-conflict queries, decomposed into five sub-behaviors (emotional validation, indirect language, framing acceptance, moral endorsement, and passive framing). Evaluated across eleven models against human baselines.
Expanding on what we missed with sycophancy
OpenAI's detailed post-mortem of the April 25, 2025 GPT-4o update that made ChatGPT noticeably sycophantic — validating doubts, fueling anger, urging impulsive actions, and reinforcing negative emotions — and was rolled back by April 28. The post explains how combined reward-signal changes (including thumbs-up/down user feedback) produced the regression, why offline evaluations and A/B tests failed to catch it, and what process changes followed, including treating model behavior issues as launch-blocking.
SycEval: Evaluating LLM Sycophancy
A framework for quantifying progressive and regressive sycophancy in LLMs (GPT-4o, Claude-Sonnet, Gemini-1.5-Pro) across math (AMPS) and medical (MedQuad) tasks under user rebuttal pressure. It measures how often models change correct answers when challenged.
Towards Understanding Sycophancy in Language Models
Demonstrates that five state-of-the-art AI assistants consistently exhibit sycophancy — matching a user's stated belief over the truthful answer — across varied free-form tasks. Traces the behaviour in part to human preference data, showing both humans and preference models non-negligibly favour convincingly-written sycophantic responses over correct ones.