K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

Key Findings

• — Synthetic patient conversations showed substantial distributional overlap with real human-AI conversations
• — The frozen GPT-4o judge reached 94.2% exact agreement with clinician consensus across 6,751 eligible item comparisons from 151 clinician-rated transcripts
• — Leading models combined strong supportive conversation with combined-risk scores above 95, while risk exploration exposed substantial variation among lower-performing configurations
• — Therapeutic prompting produced configuration-specific gains concentrated among weaker models; elevated reasoning settings produced no average improvement
• — The benchmark keeps its operational test materials protected and publishes a continuously updated leaderboard

Methodology Notes

125 model configurations (33 base models, 14 providers) evaluated on 200 multi-turn synthetic vignettes across suicide, self-harm, domestic violence, substance misuse and no-risk presentations; frozen GPT-4o judge calibrated on 151 clinician-rated transcripts (6,751 item comparisons); protected test set, so external replication of the exact evaluation is not possible. Kivira Health is a commercial mental-health AI company and several authors are affiliated with it. Preprint (v1 2026-09-14, v2 2026-09-15; announced in the 15 and 16 September cs.CL listings); no venue stated. The benchmark website existed from August 2026 without a paper; this preprint is the first document. Curator read the abstract page, the PDF title block and the leaderboard site on 2026-09-17. Update 2026-09-24: the public leaderboard is live on the site home page (bundle leaderboard-20260823T091856Z, generated 2026-08-23, served as site_data.json; 125 configurations, 200 vignettes, 20 turns); the top ten scores span 98.96 (gpt-5.5, default prompt, low reasoning) to 98.57 (claude-fable-5), a 0.4-point spread.

Sources