Skip to main content
Preprint Preliminary — Early preprints, credible essays, unreviewed grey literature

Rating Chatbot Responses to Mental Health Prompts: Rubric Development and Cross-Sectional Evaluation Study (Preprint)

Two clinicians developed a rubric of 4 to 8 binary safety and competence criteria for each of 15 Danish-language mental health prompts spanning paranoid delusion, elevated mood, suicidal ideation, depression and anxiety. They then rated 299 responses from five therapy-focused and five general-purpose chatbots, blind to chatbot identity.

Publisher

JMIR Preprints

Published

6 Nov 2026

Added

5 days ago

Key Findings

  • Interrater reliability was substantial (kappa 0.80, 95% CI 0.77 to 0.83) over 1,696 criterion ratings.
  • 5.0% of responses met all safety and competence criteria and 57.5% met neither; 21.7% were safe but not competent and 15.7% were competent but not safe.
  • Paranoid delusion prompts produced the least safe responses (3.3% met all safety criteria) and suicidal ideation the least competent (0% met all competence criteria).
  • Therapy-focused chatbots directed users to support in 6.2% of applicable responses versus 60.0% for general-purpose chatbots, while providing less harmful information. Individual chatbots passed between 40.6% and 65.9% of all criteria.

Methodology Notes

Preprint posted 2026-11-06 (Crossref record created 2026-11-11 for the JMIR Preprints DOI); not peer-reviewed. The JMIR host returns an empty page to automated clients, so title, authors, dates and the full abstract were read from the Crossref record (subtype preprint) and figures could not be checked against the rubric appendices. Two raters, consensus or expert adjudication, two responses per chatbot per prompt. The authors state that standardised rubric validity and automated rating remain to be tested. Dataset available on request per the preprint.

Authors

Benjamin Arnfred, Celia Faye Jacobsen, Christian Hardmeier, Stig Poulsen, Ulrik Nykjaer Jeppesen, Klara Coello, Annette Erlangsen, Sebastian Simonsen, Louise Birkedal Glenthoej

Tags

danishclinician-rubricjmir-preprintsmultilingual

Cite This

APA

Benjamin Arnfred et al. (2026). Rating Chatbot Responses to Mental Health Prompts: Rubric Development and Cross-Sectional Evaluation Study (Preprint). JMIR Preprints. https://preprints.jmir.org/preprint/116341