Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy
Large-scale factorial study of medical sycophancy — models abandoning correct medical answers under user pushback — crossing four conversational factors with five open-weight models over 500 MedQuAD-derived questions, totalling about 1.2 million trials. Finds conversational context, not model identity, is the dominant driver of sycophantic concession.
Publisher
arXiv (Virginia Tech)
Published
2 Aug 2026
Added
2 weeks ago
Key Findings
- Sycophancy rates vary about 67-fold across questions but only about 3-fold across models, indicating a single per-model sycophancy rate obscures the dominant sources of variance
- Fabricated authority sources roughly double concession when presented alongside the question but halve it when introduced after the model has already answered
- Chain-of-thought traces show models that re-examine their own prior answer tend to concede, while models that reason about the underlying medical facts hold their answers
Methodology Notes
Fully crossed factorial design: four conversational factors × five open-weight models × 500 MedQuAD-derived questions (~1.2M trials), with chain-of-thought analysis of concession reasoning. Preprint (arXiv 2608.01017, v1 2026-08-02), not yet peer-reviewed; title, authors, and date verified via the arXiv API.
Sources
arXiv abstract (primary)
Archived snapshot (Wayback Machine) — preserved against link rot
Authors
Kaike Ping, Buse Çarık, Caleb Wohn, Xiaohan Ding, Tongshuai Wang, Eugenia Rho
Tags
Cite This
APA
Kaike Ping et al. (2026). Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy. arXiv (Virginia Tech). https://arxiv.org/abs/2608.01017