TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots
Framework that evaluates and trains language-model therapy chatbots on two pillars: fidelity to cognitive behavioural therapy, scored with the Cognitive Therapy Rating Scale (CTRS), and therapy-specific safety, scored with four binary session-level flags (medical opinion or medication advice, failure to address crisis and imminent risk, failure to address abuse, failure to address functional impairment). It releases TherapyJudgeBench, 116 synthetic patient-therapist dialogues rated by licensed CBT-trained practitioners, to calibrate language-model judges, and uses CTRS and safety rewards to fine-tune a small therapist model with GRPO against simulated patients.
Publisher
Stanford University, University of Hong Kong and UC Berkeley authors (arXiv; NeurIPS 2026 Evaluations and Datasets Track)
Published
23 Feb 2026
Added
today
Key Findings
- TherapyJudgeBench holds 116 dialogues with 993 expert ratings; 20% of the dataset was double-annotated, and Krippendorff's alpha across CTRS skills averages 0.52 (median 0.55, range 0.23 to 0.72), with average Spearman rho 0.58 and Pearson r 0.60
- Applied without recalibration to 133 real clinician-led sessions from the AnnoMI dataset (110 externally labelled high quality, 23 low quality), the judge separates the groups on overall CTRS (mean 0.41 versus 0.13), and the medical-advice violation rate rises from 0.07 to 0.48 in the low-quality group
- GRPO fine-tuning raises human-rated average CTRS from 0.15 to 0.57 (language-model judge 0.16 to 0.59) and lowers the human-judged average safety-violation rate from 0.38 to 0.20
- Removing the safety penalty during GRPO raises the safety-violation rate (0.13 with the penalty against 0.43 without, language-model judge)
Methodology Notes
All benchmark dialogues are synthetic (a patient simulator conditioned on fictional patient profiles; no real patient data) and short. Two CTRS dimensions with low agreement were dropped from the reward. Safety labels are binary and session-level. The arXiv abstract field still carries the version 1 figures; the figures above are from the version 2 PDF (revised 3 November 2026, accepted to the NeurIPS 2026 Evaluations and Datasets Track). The dataset (Hugging Face therapygym/therapyjudgebench, CC BY 4.0) was released on 3 November 2026. Published date is the version 1 submission date. Figures re-checked against the PDF text.
Sources
arXiv paper (v2)(opens in a new tab) (primary)
TherapyJudgeBench v2.0.0 dataset (CC BY 4.0)(opens in a new tab) (3 Nov 2026)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Topics
Authors
Fangrui Huang, Souhad Chbeir, Arpandeep Khatua, Sheng Wang, Sijun Tan, Kenan Ye, Lily Bailey, Merryn Daniel, Ryan Louie, Sanmi Koyejo, Ehsan Adeli
Tags
Cite This
APA
Fangrui Huang et al. (2026). TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots. Stanford University, University of Hong Kong and UC Berkeley authors (arXiv; NeurIPS 2026 Evaluations and Datasets Track). https://arxiv.org/abs/2603.18008