Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
Decomposes sycophancy (abandoning a correct answer or endorsing a user's position under pushback) into task, model and pressure factors using 103,939 graded replies from eight language models with reasoning disabled plus two of them at maximum reasoning, each facing 200 items, 13 pressure conditions and four-turn conversations. Every reply was labelled by two frontier judges from different vendors. The dominant factor is how costly it is for the model to verify the user's claim and whether a trained guardrail covers it; the pressure tactic explains almost nothing.
Publisher
arXiv (OASIS Lab, University of California, Los Angeles; University of California, Berkeley)
Published
30 Sept 2026
Added
today
DOI
—
Key Findings
- Removing the task factor from a logistic model costs 0.485 of McFadden R-squared, against 0.139 for model family and 0.009 for pressure tactic.
- Adoption of the pushed answer is 1.3% for anchored facts and 6.4% for shallow puzzles, rising to 29.7% for deep puzzles; personal choices are endorsed in 77.0% of conversations.
- On deep puzzles, models concede in 35.5% of conversations on items they cannot solve but only 3.5% on items they can; at maximum reasoning neither tested model concedes on reasoning items.
- Fallacious or emotional framing of the pushback adds nothing beyond plain repetition.
- Three human annotators agreed with the two judges' consensus on 118 of 120 calibration items.
Methodology Notes
Models: GLM-5.2, DeepSeek-V4-Flash, Qwen3.5-397B, MiniMax-M3, Gemini-3-Flash, Grok-4.3, Claude Sonnet 5, GPT-5.6-Luna (reasoning disabled), with Claude Sonnet 5 and GPT-5.6-Luna re-run at maximum reasoning; judges Claude Opus 5 and GPT-5.6-Sol at maximum reasoning; results reported to hold under each judge alone and on 149 items shared with an earlier study. Scripted pushback, no human users. Submitted 2026-09-30 and first announced in the 8 October 2026 listing; not peer reviewed. Verified on the arXiv abstract page and the HTML full text (affiliations, model list and every number read from the text).
Sources
arXiv preprint(opens in a new tab) (primary)
HTML full text (v1)(opens in a new tab) (8 Oct 2026)
Authors
Guang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian
Tags
Cite This
APA
Guang Yang et al. (2026). Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding. arXiv (OASIS Lab, University of California, Los Angeles; University of California, Berkeley). https://arxiv.org/abs/2610.08840
Related Insights
Affective Context Amplifies Sycophancy in LLM Responses
arXiv (preprint) · 21 Aug 2026
Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
arXiv (Texas A&M University; University of Cincinnati) · 8 Sept 2026
FIGS: Evaluating Multi-Turn Sycophancy Without Penalizing Empathy
arXiv (ELLIS Institute Tübingen; Max Planck Institute for Intelligent Systems; Tübingen AI Center) · 30 Sept 2026