LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions
Characterises how people treat large language models as oracles, all-knowing authorities on subjective personal questions, and measures that reliance at scale. Applying a typology and LLM-based classifiers to 68,000 public prompts from WildChat and ThoughtTrace, the authors find oracle use rising from 2023 to 2026 and more prevalent among younger users; a privacy-preserving data-donation tool applied to 140,000 prompts from 52 participants shows the same trends. People were often unaware of their own oracle use and expressed dissatisfaction once shown it, and the paper identifies users' perceptions of AI and the models' own behaviour as the two drivers.
Publisher
arXiv (Stanford University and collaborators; affiliations not printed on the abstract page)
Published
13 Sept 2026
Added
today
Key Findings
- Applied to 68,000 public prompts (WildChat and ThoughtTrace), LLM-as-oracle use increased over 2023 to 2026 and was more prevalent among younger users.
- Longitudinal donated data (140,000 prompts from 52 participants, collected through a privacy-preserving tool) showed similar trends.
- Participants were often unaware of their own oracle use and expressed dissatisfaction with the behaviour after seeing the tool's analysis.
- Two drivers are identified, people's perceptions of AI and the behaviour of the models themselves, motivating interventions that support users' own deliberation.
Methodology Notes
Typology development, LLM-based measurement of oracle use in public usage logs, and a longitudinal data-donation study (n = 52); motivated explicitly by risks to autonomy and wellbeing. arXiv v1 submitted 2026-09-13, readable at the abstract page but not yet announced in a daily listing and with no PDF served at the time of entry, so affiliations are not confirmed beyond the authors' public affiliations (Stanford NLP group and collaborators). Same first author as the held ELEPHANT social-sycophancy row.
Sources
Topics
Authors
Myra Cheng, Lujain Ibrahim, Grace Liu, Michelle S. Lam, Vishakh Padmakumar, Nick Madibekov, Diyi Yang, Dan Jurafsky
Tags
Cite This
APA
Myra Cheng et al. (2026). LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions. arXiv (Stanford University and collaborators; affiliations not printed on the abstract page). https://arxiv.org/abs/2609.14849
Related Insights
ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs
arXiv (Stanford-led) · 20 May 2025
Enabling independent research on how people use Claude
Anthropic (Societal Impacts), with Stanford SALT Lab, Oxford Human Information Processing Lab and METR · 26 Aug 2026
The dilemmas of delegation: An analysis of policy challenges posed by Advanced AI Assistants and natural-language AI agents
Ada Lovelace Institute · 11 Nov 2025