MindEval: Benchmarking Language Models on Multi-turn Mental Health Support

Key Findings

• — All 12 evaluated LLMs average below 4 of 6 on the clinician-designed rubric
• — Particular weaknesses in problematic AI-specific patterns of communication (sycophancy, over-validation, reinforcement of maladaptive beliefs)
• — Reasoning capability and model scale do not guarantee better performance
• — Performance deteriorates as interactions lengthen and when the simulated patient has severe symptoms
• — Simulated-patient realism validated against human text; automatic scores correlate strongly with human expert judgments
• — Code, prompts and human evaluation data released (github.com/SWORDHealth/mind-eval; default 10-turn interactions)

Methodology Notes

Patient-simulation plus LLM-judge framework co-designed with licensed clinical psychologists; 12 LLMs evaluated; fully automated and model-agnostic by design, so results rest on simulated patients and LLM judging (human evaluation data released for validation). Authored by the vendor Sword Health's research team. arXiv v1 23 Nov 2025, v2 25 Nov 2025, v3 (current) 5 Dec 2025. No version of record found (Crossref title query no match; OpenAlex holds only the arXiv record). Official code at github.com/SWORDHealth/mind-eval (last push 2025-12-05). Verified at the arXiv abstract page (HTTP 200; title, six authors and version history matched); a coverage miss carried on the watchlist since 2026-08-13.

Sources