MentalHealthBench: An Expert-Informed Benchmark of AI Capabilities in Realistic Mental Health Conversations

Key Findings

• — Task-clipped scores: GPT-6 Astra 57.3, GPT-6 Sol 53.9, Claude Opus 5.5 52.4, GPT-6 Luna 50.2, Muse Spark 1.3 48.6, GPT-5.6 Sol 47.0, Claude Fable 5.1 46.4, Claude Sonnet 5 44.5, Grok 4.7 41.3, Gemini 3.8 Flash 35.5, GPT-4o (March 2025) 32.1, Gemini 3.1 Pro 32.1, Gemini 2.5 Pro 29.5.
• — Composition: 53.5% non-acute, 18.2% high-acuity, 28.3% emergency conversations; 68.1% adult, 21.2% teen, 5.8% clinician, 4.9% caregiver profiles; 159 suicide and self-harm and 115 psychosis conversations; 312 non-English conversations led by Spanish (105) and Hindi (54).
• — GPT-4o (March 2025) scores 36.5 on non-acute but 24.1 on emergency conversations, while GPT-6 Astra holds 56.6, 57.9 and 58.3 across the three acuity levels.
• — On the teen subset (age stated via a system message) Claude Opus 5.5 scores highest at 57.0, ahead of GPT-6 Astra (55.9); Grok 4.7 scores 40.5 and Gemini 2.5 Pro 29.1.
• — Rubrics written by 44 adult users of AI for emotional support align with expert rubrics on 25.7% of weight; 39.1% of weight is expert-only (mostly guardrails), 34.2% user-only (actionable next steps and tone) and 1.0% directly contradictory.
• — Context-seeking (66.5% of rubric weight) discriminates most between models; empathy/support is nearly flat across models; older models score poorly on the reality-testing axis.

Methodology Notes

Synthetic conversation prefixes generated to match privacy-preserving summaries of real ChatGPT mental-health usage; rubrics written independently by two licensed clinicians per conversation and finalised by a third; criteria weighted from -10 to +10 and classified by an LLM into ten behavioural axes. Responses sampled four times per task at default API settings and graded item by item by GPT-5.6 Sol at high reasoning effort (an OpenAI model grading OpenAI and competitor models). Effectively single-turn (the model answers the final user turn); language comparisons are descriptive; user-rubric collection was limited to non-acute conversations; chat modality only. Dataset released as a zip with a canary string, with a request not to post examples online. Verified by fetching the paper PDF (HTTP 200, 32 pages, CreationDate 2026-09-23) and the dataset zip (HTTP 200, 2.1 MB); the announcement page was read via a Wayback raw capture because openai.com returns 403 to fetchers.

Sources