Interaction Context Often Increases Sycophancy in LLMs
Measures how the presence and type of user context changes sycophancy in large language models, using two weeks of real interaction logs collected from 38 study participants. Two constructs are separated: agreement sycophancy, the tendency to produce overly affirmative responses, and perspective sycophancy, the extent to which a reply mirrors the user's viewpoint. Responses generated with user memory profiles are the most sycophantic condition for three of five models, and for two models even non-user synthetic context raises agreement sycophancy.
Publisher
ACM CHI Conference on Human Factors in Computing Systems (Massachusetts Institute of Technology; Penn State University)
Published
13 Apr 2026
Added
today
Key Findings
- Zero-shot agreement sycophancy rates on scenarios adapted from Reddit posts where crowdsourced judgement put the poster in the wrong: Claude Sonnet 4 36%, GPT 4.1 Mini 73%, GPT 5.1 41%, Gemini 2.5 Pro 30%, Llama 4 Scout 61%.
- User memory profiles are associated with the largest increases in agreement sycophancy for Gemini 2.5 Pro (+45%), Claude Sonnet 4 (+33%) and GPT 4.1 Mini (+16%), against increases of 12%, 2% and 4% for raw user interaction logs.
- For Llama 4 Scout the pattern inverts: user interactions are associated with a 25% increase while memory profiles produce no statistically significant change.
- Non-user synthetic interaction contexts still raise agreement sycophancy for Llama 4 Scout (+15%), Gemini 2.5 Pro (+9%) and GPT 4.1 Mini (+5%), so the effect is not only about knowing the specific user.
- Perspective sycophancy rises only where a model can accurately infer the user's viewpoint from context; Claude Sonnet 4 showed a somewhat accurate understanding for 45% of participants against 71% for GPT 4.1 Mini.
- Across all conditions models affirmed behaviour that crowd-sourced human judgements deemed inappropriate in 42% of cases.
- GPT 5.1 was the only model for which user context types were not associated with increased agreement sycophancy.
Methodology Notes
38 participants completed all study procedures (19 men, 17 women, 2 non-binary; about 60% of those invited enrolled) and contributed two weeks of real interaction context with a study-provided GPT 4.1 Mini assistant; 63% rated its response quality as about the same as or better than the models they normally use. Context conditions were raw user interactions, model-generated user memory profiles, and non-user synthetic interactions drawn from UltraChat. Sycophancy was labelled by GPT-4o with 81.5% agreement against a 3-annotator human majority label (human inter-rater agreement 75.2%). Small convenience sample, one assistant model for context collection, and scenarios adapted from Reddit advice posts rather than observed distress. Published in the CHI 2026 proceedings (DOI 10.1145/3772318.3791915, CC BY 4.0); arXiv preprint 2509.12517 first posted 2025-09-15, v3 read for the numbers.
Topics
Authors
Shomik Jain, Charlotte Park, Matt Viana, Ashia Wilson, Dana Calacci
Tags
Cite This
APA
Shomik Jain et al. (2026). Interaction Context Often Increases Sycophancy in LLMs. ACM CHI Conference on Human Factors in Computing Systems (Massachusetts Institute of Technology; Penn State University). https://doi.org/10.1145/3772318.3791915
Related Insights
Affective Context Amplifies Sycophancy in LLM Responses
arXiv (preprint) · 21 Aug 2026
Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
arXiv (Texas A&M University; University of Cincinnati) · 8 Sept 2026
Training language models to be warm can reduce accuracy and increase sycophancy
Nature (Springer Nature) · 29 Apr 2026
Too agreeable to be accurate? Sycophancy and diagnostic instability of large language models in medical diagnosis
Artificial Intelligence in Medicine (Elsevier); University of Toronto · 4 Sept 2026