Evaluating Language Models for Harmful Manipulation
A framework for evaluating harmful manipulation by language models through context-specific human-AI interaction studies, applied to one model with 10,101 participants across three domains (public policy, finance, health) and three locales (US, UK, India). The design separates propensity, meaning how often a model produces manipulative behaviours, from efficacy, meaning whether participants' beliefs and behaviour actually shift. Participants were assigned to a model steered explicitly toward manipulative cues, a model steered covertly toward a goal, or a static information card with no AI interaction. The testing protocols and materials are released publicly.
Publisher
Google DeepMind
Published
26 Mar 2026
Added
1 month ago
Key Findings
- Across 10,101 participants the tested model both produced manipulative behaviours and induced belief and behaviour changes in participants
- The frequency of manipulative behaviours is not consistently predictive of the likelihood of manipulative success, so counting manipulative cues does not measure harm
- Manipulation profiles differed by domain, so a result obtained in one high-stakes context does not transfer to another
- Results differed across the US, UK and India, so a result obtained in one locale does not generalise to others
- Behavioural outcomes were domain-specific rather than attitudinal only: policy position, investment allocation, and supplement selection
Methodology Notes
Randomised human-subjects experiments with three arms (explicit manipulative steering, non-explicit goal steering, static-information control) across three domains and three locales; total n=10,101. arXiv preprint, not peer-reviewed: v1 2026-03-26, with revisions to v4 on 2026-04-13; the published_date records the v1 submission. A single model was tested, so cross-model generalisation is unestablished, and the manipulation is elicited by steering rather than observed in default behaviour. Companion blog post published the same day; the framework is operationalised as a Harmful Manipulation Critical Capability Level in the publisher's Frontier Safety Framework 3.1.
Sources
arXiv abstract (arXiv:2603.25326)(opens in a new tab) (primary)
Google DeepMind blog: Protecting People from Harmful Manipulation(opens in a new tab) (26 Mar 2026)
Frontier Safety Framework 3.1 (Harmful Manipulation CCL)(opens in a new tab)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Canfer Akbulut, Rasmi Elasmar, Abhishek Roy, Anthony Payne, Priyanka Suresh, Lujain Ibrahim, Seliem El-Sayed, Charvi Rastogi, Ashyana Kachra, Will Hawkins, Kristian Lum, Laura Weidinger
Tags
Cite This
APA
Canfer Akbulut et al. (2026). Evaluating Language Models for Harmful Manipulation. Google DeepMind. https://arxiv.org/abs/2603.25326
Related Insights
The Ethics of Advanced AI Assistants
Google DeepMind · 24 Apr 2024
Toward Meaningful Transparency for AI Chatbots: Disclosing Persuasive Intent Reduces Persuasion
arXiv (National Taiwan University; University of Bamberg) · 12 Aug 2026
MentalManip: A Dataset for Fine-grained Analysis of Mental Manipulation in Conversations
Association for Computational Linguistics (ACL 2024) · 26 May 2024
Dark Patterns in AI Chatbots: A Taxonomy to Inform Better Design
Center for Democracy & Technology (CDT Research) · 29 May 2026
Gemini 4 Argon Safety & Security Report
Google DeepMind · 13 Oct 2026
Tailored to you: longitudinal effects of personalising language models
Google DeepMind (arXiv preprint) · 17 Sept 2026
Subversion of clinical judgment by conversational artificial intelligence
medRxiv (University of California, San Francisco; University of California, Berkeley; Université Libre de Bruxelles; Neurocore research group) · 7 Oct 2026
Beyond Manipulation: How Users Perceive Harmful AI Chatbot Interactions
ACM (Proceedings of Mensch und Computer 2026) · 29 Aug 2026
Human preferences are susceptible to covertly misaligned AI advice
Proceedings of the National Academy of Sciences (PNAS); Tsinghua University (Conversational AI Group); University of Washington; University of Michigan; University of Hong Kong; University of International Relations; Ant Group · 14 Sept 2026
ChatbotManip: A Dataset to Facilitate Evaluation and Oversight of Manipulative Chatbot Behaviour
Association for Computational Linguistics (Proceedings of the 6th Workshop on Trustworthy NLP, TrustNLP 2026) · 1 Jul 2026