Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling
Proposes PCSA (Persona-based Client Simulation Attack), a red-teaming framework that simulates coherent, persona-driven counselling clients to probe LLM safety alignment. Across seven LLMs it elicited unauthorised medical advice, delusion reinforcement, and implicit encouragement of risky actions.
Publisher
arXiv
Published
6 Apr 2026
Added
3 months ago
DOI
—
Key Findings
- Persona-driven simulated clients surface safety failures that generic prompting misses
- Elicited unauthorised medical advice, delusion reinforcement, and implicit encouragement of risky actions across seven LLMs
- Frames therapeutic-interaction harms as distinct from generic jailbreak payloads
Methodology Notes
Preprint (arXiv 2604.04842, v1 6 April 2026; v2 posted 5 September 2026 with the comment 'EMNLP 2026 Findings', so a Findings of EMNLP 2026 version of record is expected in the Anthology in November 2026). Red-teaming framework evaluated across seven LLMs in simulated counselling dialogues.
Sources
arXiv abstract(opens in a new tab) (primary)
EMNLP 2026 programme (paper 6875-FIND, Findings of EMNLP 2026, poster 25 Oct 2026)(opens in a new tab) (29 Sept 2026)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Qingyang Xu, Yaling Shen, Stephanie Fong, Zimu Wang, Yiwen Jiang, Xiangyu Zhao, Jiahe Liu, Zhongxing Xu, Vincent Lee, Zongyuan Ge
Tags
Cite This
APA
Qingyang Xu et al. (2026). Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling. arXiv. https://arxiv.org/abs/2604.04842
Related Insights
The Slow Drift of Support: Boundary Failures in Multi-Turn Mental Health LLM Dialogues
arXiv · 2 Jan 2026
The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models
arXiv (King's College London-led) · 13 Sept 2025