ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs
A benchmark measuring 'social sycophancy' — excessive preservation of a user's self-image or 'face' — across advice and moral-conflict queries, decomposed into five sub-behaviors (emotional validation, indirect language, framing acceptance, moral endorsement, and passive framing). Evaluated across eleven models against human baselines.
Publisher
arXiv (Stanford-led)
Published
20 May 2025
Added
3 months ago
Key Findings
- LLMs preserved user 'face' roughly 45 percentage points more than humans across queries
- Models affirmed both sides of a moral conflict in about 48% of cases
- Social sycophancy is measurable and pervasive beyond simple factual agreement
Methodology Notes
Preprint (arXiv, 2025-05-20). Introduces the ELEPHANT metric suite over advice-seeking and moral-dilemma datasets with human comparison; measures behavior on curated prompts rather than live user harm.
Sources
arXiv abstract(opens in a new tab) (primary)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, Dan Jurafsky
Tags
Cite This
APA
Myra Cheng et al. (2025). ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs. arXiv (Stanford-led). https://arxiv.org/abs/2505.13995
Related Insights
Expanding on what we missed with sycophancy
OpenAI · 2 May 2025
SycEval: Evaluating LLM Sycophancy
arXiv (Stanford-led) · 12 Feb 2025
How people ask Claude for personal guidance
Anthropic · 30 Apr 2026
Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs
Association for Computational Linguistics (ACL 2026 Long Papers); Stanford University · 1 Jul 2026
Towards Understanding Sycophancy in Language Models
Anthropic · 20 Oct 2023
MentalManip: A Dataset for Fine-grained Analysis of Mental Manipulation in Conversations
Association for Computational Linguistics (ACL 2024) · 26 May 2024
Feeling Right vs. Being Right: How AI Sycophancy Affects Value-Laden Deliberation
Association for Computational Linguistics (Proceedings of ACL 2026, Long Papers); Seoul National University; Taejae University; Yonsei University · 1 Jul 2026
Affective Context Amplifies Sycophancy in LLM Responses
arXiv (preprint) · 21 Aug 2026
Faithful Where It Can Be Checked: Auditing a Reflection Agent Against Its System Prompt in a Randomized Trial
arXiv (Stanford University; University of Virginia; NAVER Cloud) · 17 Sept 2026
FIGS: Evaluating Multi-Turn Sycophancy Without Penalizing Empathy
arXiv (ELLIS Institute Tübingen; Max Planck Institute for Intelligent Systems; Tübingen AI Center) · 30 Sept 2026
LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions
arXiv (Stanford University and collaborators; affiliations not printed on the abstract page) · 13 Sept 2026
Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversation
arXiv (Hong Kong University of Science and Technology (Guangzhou); Chinese University of Hong Kong, Shenzhen; Dongbei University of Finance and Economics) · 3 Sept 2026
Mitigating Social Sycophancy via Pluralistic Preference Optimization
arXiv (Stanford University; University of Washington; Amazon) · 1 Oct 2026
Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in Language Models
arXiv (Harvard Kennedy School; Harvard Department of Statistics; Stanford University) · 22 Sept 2026
API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
arXiv (Stanford University) · 8 Sept 2026
What Users Cannot See: Evaluating LLM Emotional Support Beyond User Preference
arXiv (University of Pennsylvania, Computer and Information Science; Stony Brook University; Vanderbilt University) · 20 May 2026
Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy
arXiv (Virginia Tech) · 2 Aug 2026
Dark Patterns in AI Chatbots: A Taxonomy to Inform Better Design
Center for Democracy & Technology (CDT Research) · 29 May 2026
The Influence of Patient Persona and Affective Framing on Management Recommendations of ChatGPT, Gemini and Claude for Unruptured Intracranial Aneurysms: A Comparative Benchmarking Study
Neurosurgical Review (Springer); Monash Health Department of Neurosurgery; Monash University · 3 Aug 2026
The Adaptation Dilemma: Cultural Fit Does Not Guarantee Safety in Mental-Health LLMs
PsyArXiv (OSF); McGill University (Department of Psychiatry; Department of Philosophy); The Decision Lab · 16 Sept 2026
Large Language Models Are More Sycophantic in Chinese Than in English
PsyArXiv · 7 Oct 2026
Sycophantic AI decreases prosocial intentions and promotes dependence
Science (AAAS) · 26 Mar 2026