Rating Chatbot Responses to Mental Health Prompts: Rubric Development and Cross-Sectional Evaluation Study (Preprint)
Two clinicians developed a rubric of 4 to 8 binary safety and competence criteria for each of 15 Danish-language mental health prompts spanning paranoid delusion, elevated mood, suicidal ideation, depression and anxiety. They then rated 299 responses from five therapy-focused and five general-purpose chatbots, blind to chatbot identity.
Publisher
JMIR Preprints
Published
6 Nov 2026
Added
5 days ago
Key Findings
- Interrater reliability was substantial (kappa 0.80, 95% CI 0.77 to 0.83) over 1,696 criterion ratings.
- 5.0% of responses met all safety and competence criteria and 57.5% met neither; 21.7% were safe but not competent and 15.7% were competent but not safe.
- Paranoid delusion prompts produced the least safe responses (3.3% met all safety criteria) and suicidal ideation the least competent (0% met all competence criteria).
- Therapy-focused chatbots directed users to support in 6.2% of applicable responses versus 60.0% for general-purpose chatbots, while providing less harmful information. Individual chatbots passed between 40.6% and 65.9% of all criteria.
Methodology Notes
Preprint posted 2026-11-06 (Crossref record created 2026-11-11 for the JMIR Preprints DOI); not peer-reviewed. The JMIR host returns an empty page to automated clients, so title, authors, dates and the full abstract were read from the Crossref record (subtype preprint) and figures could not be checked against the rubric appendices. Two raters, consensus or expert adjudication, two responses per chatbot per prompt. The authors state that standardised rubric validity and automated rating remain to be tested. Dataset available on request per the preprint.
Sources
JMIR Preprints(opens in a new tab) (primary)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Topics
Authors
Benjamin Arnfred, Celia Faye Jacobsen, Christian Hardmeier, Stig Poulsen, Ulrik Nykjaer Jeppesen, Klara Coello, Annette Erlangsen, Sebastian Simonsen, Louise Birkedal Glenthoej
Tags
Cite This
APA
Benjamin Arnfred et al. (2026). Rating Chatbot Responses to Mental Health Prompts: Rubric Development and Cross-Sectional Evaluation Study (Preprint). JMIR Preprints. https://preprints.jmir.org/preprint/116341