SycEval: Evaluating LLM Sycophancy
A framework for quantifying progressive and regressive sycophancy in LLMs (GPT-4o, Claude-Sonnet, Gemini-1.5-Pro) across math (AMPS) and medical (MedQuad) tasks under user rebuttal pressure. It measures how often models change correct answers when challenged.
Publisher
arXiv (Stanford-led)
Published
12 Feb 2025
Added
3 months ago
Key Findings
- Sycophancy occurred in 58.2% of cases, with 78.5% persistence across turns
- Preemptive rebuttals triggered more sycophancy than in-context rebuttals
- Distinguishes progressive (toward correct) from regressive (away from correct) sycophancy
Methodology Notes
Preprint (arXiv 2502.08177, 2025-02-12). Rebuttal-pressure protocol over math and medical QA; measures answer stability rather than emotional/relational sycophancy.
Sources
arXiv abstract(opens in a new tab) (primary)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Aaron Fanous, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, Anson Zhou, Roxana Daneshjou, Sanmi Koyejo
Tags
Cite This
APA
Aaron Fanous et al. (2025). SycEval: Evaluating LLM Sycophancy. arXiv (Stanford-led). https://arxiv.org/abs/2502.08177
Related Insights
ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs
arXiv (Stanford-led) · 20 May 2025
Expanding on what we missed with sycophancy
OpenAI · 2 May 2025
How people ask Claude for personal guidance
Anthropic · 30 Apr 2026
Towards Understanding Sycophancy in Language Models
Anthropic · 20 Oct 2023
Affective Context Amplifies Sycophancy in LLM Responses
arXiv (preprint) · 21 Aug 2026
Conduct Under Pressure: What Sixty Language Models Do When a User Pushes
arXiv (Cornell Tech) · 21 Sept 2026
EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots
arXiv (MindSurf) · 29 Jun 2026
Measuring and Detecting Harmful AI Sycophancy
arXiv preprint · 6 Aug 2026
Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
arXiv (Texas A&M University; University of Cincinnati) · 8 Sept 2026
Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy
arXiv (Virginia Tech) · 2 Aug 2026
The Doctor Will Agree With You Now: Sycophancy of Large Language Models in Multi-Turn Medical Conversations
Association for Computational Linguistics (Proceedings of the 1st Workshop on Linguistic Analysis for Health, HeaLing 2026) · 1 Mar 2026
Sycophantic AI decreases prosocial intentions and promotes dependence
Science (AAAS) · 26 Mar 2026