CrisisBench: an evaluation suite for crisis counseling AI
Preprint introducing a privacy-preserving benchmark suite for LLMs in crisis counselling, built from conversation text and metadata of the Vandrevala Foundation's WhatsApp crisis helpline in India. It has a single-turn set of 1,085 tasks from 200 de-identified conversations and a multi-turn set of 1,185 simulated patients derived from hotline metadata, scored with a 23-item conversation-level rubric and an LLM judge calibrated against expert counsellors. The systems evaluated are counsellor-facing copilots.
Publisher
Stanford University, Vandrevala Foundation and collaborating institutions (arXiv preprint)
Published
11 Sept 2026
Added
2 days ago
DOI
—
Key Findings
- The 23-item rubric reached human-human agreement of Krippendorff's alpha 0.51 and LLM judge-human agreement of alpha 0.54 (Pearson r 0.80).
- On single-turn tasks, all evaluated systems (GPT-4.1, GPT-4.1-mini, GPT-5 nano, GPT-5 mini, GPT-5.2, plus draft-then-edit and retrieval-augmented variants) underperformed human counsellors on excessive empathy and excessive questioning; the paper also reports a missed-risk failure rate for the tested systems.
- On multi-turn evaluation, retrieval-augmented generation scored highest overall (0.961) with all seven systems within a 0.058 band; microskills and style were at ceiling (0.994 to 1.000), and productivity (0.869 to 0.988) and safety and risk items drove most of the variation.
- Counsellors rated the simulated patients realistic (86%) and useful (92%).
Methodology Notes
Preprint, not peer-reviewed. The curator read the arXiv abstract page and the 51-page PDF text. The listing shows a submission date of 11 September 2026 although the identifier prefix is 2611; the abstract-page date is used. Data come from one helpline in one country, and judge reliability is moderate (alpha 0.54). The authors state that rubrics, judge prompts and tasks are released and that the suite is incorporated into MedHELM; the curator did not inspect the released files.
Sources
Authors
Akshay Swaminathan, Filip Ramazan, Sharang Phadke, Kevina Wang, Ehsan Adeli, Nigam H. Shah
Tags
Cite This
APA
Akshay Swaminathan et al. (2026). CrisisBench: an evaluation suite for crisis counseling AI. Stanford University, Vandrevala Foundation and collaborating institutions (arXiv preprint). https://arxiv.org/abs/2611.21622