Development, validation and AI chatbot evaluation of the Adolescent Mental Health AI Response Tool
Develops and validates AMHART, a clinician-informed instrument for auditing whether chatbot replies to adolescents contain guideline-informed content across seven domains, and applies it to four general-purpose chatbots. Multi-turn conversations with adolescent personas (varying by sex, diagnosis and symptom severity) were scored by a three-model language-model evaluator ensemble whose agreement with human reference labels was assessed.
Publisher
Nature Mental Health (Springer Nature); RAND, Harvard Medical School, MIT Media Lab
Published
28 Oct 2026
Added
today
Key Findings
- The instrument audits guideline-informed response content across seven domains; candidate items were reviewed by clinicians for evidence base, objectivity and clinical importance, leaving a subset with strong content validity and inter-rater reliability
- Four chatbots were tested in multi-turn conversations with adolescent personas: ChatGPT-5, ChatGPT-4, Claude 4.5 and Gemini 2.5
- AMHART scores indicated greater coverage of applicable criteria for anxiety than depression and for moderate than severe presentations
- The journal's plain-language summary adds that the evaluation identified gaps for coping guidance
Methodology Notes
Abstract-level record: the article body is behind the Nature subscription wall and the PMC copy is withheld, so no numeric scores, rater statistics or per-chatbot results are recorded here. Abstract and dates read on the nature.com article page (online 28 October 2026) and matched to the PubMed record (PMID 42974409). The evaluator ensemble is a language-model panel, so scores measure rubric coverage in replies, not outcomes for adolescents. The article lists code and data availability sections.
Sources
Nature Mental Health article(opens in a new tab) (primary)
PubMed record(opens in a new tab) (28 Oct 2026)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Ryan K. McBain, Jonathan H. Cantor, Brooke Hollister, Nicholas Kong, Adrian Salas, Joshua Breslau, Fang Zhang, Li Ang Zhang, Alyssa Burnett, Benjamin Rader, Shannon Donofry, Jessica L. Sousa, Bradley D. Stein, Pat Pataranutaporn, Hao Yu
Tags
Cite This
APA
Ryan K. McBain et al. (2026). Development, validation and AI chatbot evaluation of the Adolescent Mental Health AI Response Tool. Nature Mental Health (Springer Nature); RAND, Harvard Medical School, MIT Media Lab. https://www.nature.com/articles/s44220-026-00753-y
Related Insights
General-purpose AI as a source of mental health support amongst 24,000 adolescents in England
npj Mental Health Research (Springer Nature); University of Oxford · 27 Oct 2026
Raising the Bar for Chinese Adolescent LLM Safety: A Culturally-Grounded, Fine-Grained Benchmark
arXiv (Southern University of Science and Technology; City University of Hong Kong; Lingnan University) · 28 Sept 2026