Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls

Key Findings

• — On the four analytical decisions, participants went with the agent's assigned lean 86% of the time, against 50% expected by chance; the registered pooled flip rate was 0.72 (95% CI 0.60 to 0.83, one-sided p<.001, n=195 decisions)
• — The shift on the final policy recommendation was not statistically significant under the registered covariate-adjusted estimator: 0.17 (95% CI -0.06 to 0.40, one-sided p=0.069); the unadjusted estimate was 0.22 (one-sided p=0.037)
• — In an exploratory in-sample analysis, the four intermediate leans alone correctly predicted 82% of rollout-or-discontinue recommendations, against a 61% majority baseline (AUC 0.84)
• — Where a participant had taken a side before the agent voiced its lean, an opposing lean made switching 53 percentage points more likely than an aligned one (95% CI 0.24 to 0.82, 38 qualifying trials); participants switched 60% of the time when the agent opposed them
• — When asked afterwards to justify their choices, 77% (37 of 48) repeated an agent talking point, sometimes nearly verbatim, while 15% mentioned the agent in any answer; 52% rated the final recommendation as more their own than the agent's
• — The agent voiced a side first on 77% of decisions and participants contested the agent's lean on 23%
• — Among 11 participants who chose to work without the agent, 6 (55%) made serious methodological errors, compared with 3 of 50 (6%) who used it; this comparison was not randomized and the no-AI group was less experienced
• — With no lean assigned, Claude Opus 4.8 made the same five choices in 91% of 123 runs

Methodology Notes

Randomized controlled trial with per-decision random assignment of the agent's lean, delivered through a hidden system prompt; participants were not told. 61 participants recruited through a paid expert-annotator marketplace after a five-part vendor screen completed the task; 11 who did not use the agent form a non-randomized comparison group; 48 of the 50 analysed completed the surveys. Single fictional task (Greenfields Farming Co-operative), one domain, one model (Claude Opus 4.8 at high effort in a web VS Code with Claude Code); median 65 minutes. Leans were deliberately strengthened, so effect sizes describe an agent that holds and voices a preference. Pre-analysis plan registered at osf.io/t3q59 after data collection had begun (17 participants had started). Internal ethics review. Decision coding by an AI model with author adjudication. Authors are Anthropic staff and an Anthropic Fellows participant; the paper is not peer-reviewed. Published as a PDF on Anthropic's CDN with an accompanying research post. Research post dated 27 October 2026; PDF metadata created 26 October 2026 (UTC).

Sources