Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling
Proposes PCSA (Persona-based Client Simulation Attack), a red-teaming framework that simulates coherent, persona-driven counselling clients to probe LLM safety alignment. Across seven LLMs it elicited unauthorised medical advice, delusion reinforcement, and implicit encouragement of risky actions.
Publisher
arXiv
Published
6 Apr 2026
Added
1 month ago
DOI
—
Key Findings
- Persona-driven simulated clients surface safety failures that generic prompting misses
- Elicited unauthorised medical advice, delusion reinforcement, and implicit encouragement of risky actions across seven LLMs
- Frames therapeutic-interaction harms as distinct from generic jailbreak payloads
Methodology Notes
Preprint (arXiv 2604.04842, v1 6 April 2026). Red-teaming framework evaluated across seven LLMs in simulated counselling dialogues.
Sources
arXiv abstract (primary)
Archived snapshot (Wayback Machine) — preserved against link rot
Authors
Qingyang Xu, Yaling Shen, Stephanie Fong, Zimu Wang, Yiwen Jiang, Xiangyu Zhao, Jiahe Liu, Zhongxing Xu, Vincent Lee, Zongyuan Ge
Tags
Cite This
APA
Qingyang Xu et al. (2026). Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling. arXiv. https://arxiv.org/abs/2604.04842
Related Insights
The Slow Drift of Support: Boundary Failures in Multi-Turn Mental Health LLM Dialogues
arXiv · 2 Jan 2026
The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models
arXiv (King's College London-led) · 13 Sept 2025