SPINE: Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
Introduces SPINE, a benchmark in which a language-model proxy plays a persistent but mistaken user and adaptively challenges a target model for up to 25 turns on 100 false-presupposition and 100 unethical-request items, with a judge scoring position strength each turn. Evaluates four production systems and three Olmo-3-7B variants and reports collapse rates as a function of conversation length, tactic and user-proxy adaptivity.
Publisher
arXiv (Texas A&M University; University of Cincinnati)
Published
8 Sept 2026
Added
1 month ago
Key Findings
- Collapse rates increase with conversation length for every post-trained model tested, from 0%-57% within 5 turns to 20%-97% within 25 turns depending on model and scenario; short-horizon protocols underestimate sycophancy.
- For models with accessible reasoning traces, the correct position remains represented in the trace in 50%-89% of analysed collapse turns, which the authors read as the model choosing to please the user rather than lacking the knowledge.
- An adaptive LLM user proxy exposes more sycophantic collapse than pre-generated scripted pushback.
- Among tactics, emotional appeals are most associated with inducing collapse, with the model's position weakening in 44.3% of turns using that tactic.
- Production systems tested are Claude Sonnet 5, GPT-5.6 Terra, Gemini 3.1 Pro and DeepSeek V4 Pro; Claude Sonnet 5 serves as the judge, with human-judge verdict agreement of 86% (false presuppositions), 90% (unethical queries) and 88% overall on a 100-turn stratified sample.
Methodology Notes
200 items (100 false presuppositions, 100 unethical queries); target models: four production systems plus three Olmo-3-7B variants; judge is Claude Sonnet 5, checked by one human annotator on a verdict-stratified sample of 100 turns (25 collapse and 25 hold per bank). No real users; the judge and one target share a developer. arXiv 2609.09090 v1 8 September 2026 (announced 9 September); v2 13 October 2026 retitled 'SPINE: ...', added numeric results to the abstract, and changed data availability from 'released' (v1) to 'Code, data, and transcripts will be released upon publication'. Not peer reviewed.
Sources
arXiv preprint(opens in a new tab) (primary)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Leyuan Tang, Kangda Wei, Tianyu Jiang, Ruihong Huang
Tags
Cite This
APA
Leyuan Tang et al. (2026). SPINE: Measuring LLM Sycophancy under Sustained Multi-Turn Pressure. arXiv (Texas A&M University; University of Cincinnati). https://arxiv.org/abs/2609.09090
Related Insights
SycEval: Evaluating LLM Sycophancy
arXiv (Stanford-led) · 12 Feb 2025
Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversation
arXiv (Hong Kong University of Science and Technology (Guangzhou); Chinese University of Hong Kong, Shenzhen; Dongbei University of Finance and Economics) · 3 Sept 2026
How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement
arXiv (University of Edinburgh; Middle East Technical University); accepted to EMNLP 2026 · 7 Sept 2026
When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressure
Association for Computational Linguistics (Proceedings of ACL 2026, Long Papers); Harbin Institute of Technology · 1 Jul 2026
Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy
Association for Computational Linguistics (ACL 2026 Long Papers); The Hong Kong Polytechnic University; HKUST · 1 Jul 2026
Reasoning Is Not All You Need: Examining LLMs for Multi-Turn Mental Health Conversations
Association for Computational Linguistics (Proceedings of ACL 2026, Long Papers); Georgia Institute of Technology · 1 Jul 2026
Too agreeable to be accurate? Sycophancy and diagnostic instability of large language models in medical diagnosis
Artificial Intelligence in Medicine (Elsevier); University of Toronto · 4 Sept 2026
Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
arXiv (OASIS Lab, University of California, Los Angeles; University of California, Berkeley) · 30 Sept 2026
FIGS: Evaluating Multi-Turn Sycophancy Without Penalizing Empathy
arXiv (ELLIS Institute Tübingen; Max Planck Institute for Intelligent Systems; Tübingen AI Center) · 30 Sept 2026
Interaction Context Often Increases Sycophancy in LLMs
ACM CHI Conference on Human Factors in Computing Systems (Massachusetts Institute of Technology; Penn State University) · 13 Apr 2026
Fallibility, persuadability, and correctability of large language models under sustained conversational misinformation pressure
Scientific Reports (Nature Portfolio); University of Arizona · 1 Sept 2026