When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation
Introduces two large-scale mental-health evaluation resources: MentalBench-100k (10,000 single-session conversations paired with nine LLM responses = 100,000 pairs) and MentalAlign-70k (70,000 ratings comparing four LLM judges against human experts on seven attributes grouped into Cognitive Support and Affective Resonance). Assesses when LLM-as-judge evaluation is reliable in mental-health contexts.
Publisher
Association for Computational Linguistics (EACL 2026)
Published
24 Mar 2026
Added
3 months ago
Key Findings
- Releases MentalBench-100k (100,000 conversation-response pairs) and MentalAlign-70k (70,000 human-vs-LLM-judge ratings)
- Evaluates LLM-as-judge reliability against human experts across seven attributes
- Groups quality attributes into Cognitive Support and Affective Resonance scores
Methodology Notes
Peer-reviewed, Proceedings of the 19th EACL (2026), Anthology ID 2026.eacl-long.180, DOI 10.18653/v1/2026.eacl-long.180 (conference 24-29 March 2026, Rabat; proceedings dated 2026). Preprint arXiv 2510.19032 (21 Oct 2025).
Sources
ACL Anthology(opens in a new tab) (primary)
arXiv preprint(opens in a new tab)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Prathiba Dhanesh, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi
Tags
Cite This
APA
Abeer Badawi et al. (2026). When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation. Association for Computational Linguistics (EACL 2026). https://aclanthology.org/2026.eacl-long.180/
Related Insights
VERA-MH: Reliability and Validity of an Open-Source AI Safety Evaluation in Mental Health
arXiv (Spring Health / Slingshot AI-affiliated author team) · 4 Feb 2026
TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health
arXiv · 3 Mar 2026
AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety
arXiv · 3 Jun 2026
MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models
Association for Computational Linguistics (Findings of ACL 2026) · 1 Jul 2026
Large Language Model–Based Chatbots and Agentic AI for Mental Health Counseling: Systematic Review of Methodologies, Evaluation Frameworks, and Ethical Safeguards
JMIR AI · 13 Mar 2026