SycEval: Evaluating LLM Sycophancy
A framework for quantifying progressive and regressive sycophancy in LLMs (GPT-4o, Claude-Sonnet, Gemini-1.5-Pro) across math (AMPS) and medical (MedQuad) tasks under user rebuttal pressure. It measures how often models change correct answers when challenged.
Key Findings
- Sycophancy occurred in 58.2% of cases, with 78.5% persistence across turns
- Preemptive rebuttals triggered more sycophancy than in-context rebuttals
- Distinguishes progressive (toward correct) from regressive (away from correct) sycophancy
Methodology Notes
Preprint (arXiv 2502.08177, 2025-02-12). Rebuttal-pressure protocol over math and medical QA; measures answer stability rather than emotional/relational sycophancy.
Sources
arXiv abstract (primary)
Archived snapshot (Wayback Machine) — preserved against link rot
Authors
Aaron Fanous, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, Anson Zhou, Roxana Daneshjou, Sanmi Koyejo
Tags
Cite This
APA
Aaron Fanous et al. (2025). SycEval: Evaluating LLM Sycophancy. arXiv (Stanford-led). https://arxiv.org/abs/2502.08177
Related Insights
ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs
arXiv (Stanford-led) · 20 May 2025
Expanding on what we missed with sycophancy
OpenAI · 2 May 2025
How people ask Claude for personal guidance
Anthropic · 30 Apr 2026
Towards Understanding Sycophancy in Language Models
Anthropic · 20 Oct 2023