Skip to main content
Peer-reviewed Authoritative — Peer-reviewed venues, standards bodies, regulators, official government publications

ChatGPT Health performance in a structured test of triage recommendations

Brief Communication reporting a structured stress test of ChatGPT Health, the consumer health feature OpenAI launched in January 2026, using 60 clinician-authored vignettes across 21 clinical domains under 16 factorial conditions (960 responses). Triage accuracy followed an inverted U across acuity, with under-triage of gold-standard emergencies and over-triage of non-urgent cases. A separate set of suicidal-ideation vignettes found the product's crisis banner fired inconsistently and less often for presentations with an identified method.

Publisher

Nature Medicine (Springer Nature); Icahn School of Medicine at Mount Sinai

Published

23 Feb 2026

Added

1 month ago

Key Findings

  • Among gold-standard emergencies, 51.6% of responses (33/64) were under-triaged to 24-48 h evaluation; 28 of the 33 were asthma-exacerbation cases
  • Accuracy was 35.2% for non-urgent and 48.4% for emergency presentations, against 93.0% for semi-urgent and 76.9% for urgent; 64.8% (83/128) of non-urgent cases were over-triaged
  • Of eight prespecified tests, only anchoring statements shifted triage (edge-case shifts 3.3% to 13.3%, OR 11.7, 95% CI 3.7-36.6); race, sex and access barriers had no significant effect
  • In a 27-year-old's overdose-ideation vignette, crisis-intervention messages appeared in 0/16 responses that included normal objective findings and 16/16 without them
  • Across 14 suicidal-ideation vignettes the 988 'Help is available' interstitial fired in 4; the other 10 produced no safety alert in any of 160 responses, and it fired more reliably when no means of self-harm was identified

Methodology Notes

Responses collected via the ChatGPT Health web interface (gpt-5-mini thinking backbone) on 9-11 January 2026, one new thread per condition, no regeneration; 2x2x2x2 within-vignette factorial (anchoring, access barrier, race, sex); gold standard from three physicians; cluster bootstrap and mixed-effects logistic regression with Holm correction. Single standardized prompt template, forced four-level answer (A monitor at home to D emergency department now); no prompt-sensitivity analysis (authors' stated limitation). Synthetic vignettes, no human participants. Data and prompts on Zenodo 10.5281/zenodo.18451491. Received 15 Jan 2026, accepted 20 Feb, published online 23 Feb 2026; print Nat Med 32(5):1671-1675. Two later documents bear on the findings and are linked as sources. A Macquarie University re-analysis (arXiv 2603.11413, not peer reviewed; v5 posted 2026-10-20 and retitled 'Evaluation format, not only model capability, shapes measured triage failure in consumer health AI') reports that much of the emergency under-triage rate depends on output format and on mapping prose onto a four-point scale, and that it did not reproduce as a stable cross-model finding when the 60 released vignettes were rerun on six API models; it did not test the ChatGPT Health product. A medRxiv preprint from the same Mount Sinai group (10.64898/2026.07.21.26358588, posted 2026-07-23) tested ChatGPT Health on 255 cases, including real emergency-department and nurse-line cases, in single-turn and multi-turn modes and reports that discordant recommendations skewed to lower acuity in both modes (70.8% and 69.0% under-triage).

Authors

Ashwin Ramaswamy, Alvira Tyagi, Hannah Hugo, Joy Jiang, Pushkala Jayaraman, Mateen Jangda, Alexis E. Te, Steven A. Kaplan, Joshua Lampert, Robert Freeman, Nicholas Gavin, Ashutosh K. Tewari, Ankit Sakhuja, Bilal Naved, Alexander W. Charney, Mahmud Omar, Michael A. Gorin, Eyal Klang, Girish N. Nadkarni

Tags

chatgpt-healthtriagenature-medicinecrisis-banner988mount-sinai

Cite This

APA

Ashwin Ramaswamy et al. (2026). ChatGPT Health performance in a structured test of triage recommendations. Nature Medicine (Springer Nature); Icahn School of Medicine at Mount Sinai. https://www.nature.com/articles/s41591-026-04297-7

Related Insights

Preprint

API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

arXiv (Stanford University) · 8 Sept 2026

Preprint

Single-turn emergency psychiatric triage across 15 frontier AI chatbots

arXiv (Max Planck UCL Centre for Computational Psychiatry and Ageing Research; UK AI Security Institute; University of Oxford; Microsoft AI) · 28 Apr 2026

Benchmark / dataset

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

arXiv (Massachusetts Institute of Technology); accepted to Findings of EMNLP 2026 · 29 Sept 2026

Preprint

How Safe Is ChatGPT Health Advice? Evidence from Real User Conversations in Four Global South Countries

arXiv (Rutgers University; independent researcher) · 13 Oct 2026

Peer-reviewed

Multidimensional evaluation of generative AI chatbots for public consultation on sudden cardiac death: safety, accuracy, empathy, reliability, quality, and readability across six models

BMC Public Health (BMC, Springer Nature); Bozhou People's Hospital · 6 Oct 2026

Preprint

AI chatbots abandon sleep apnoea referral when simulated patients minimise symptoms

medRxiv; Guy's and St Thomas' NHS Foundation Trust (Lane Fox Unit/Sleep Disorders Centre); n3 Labs · 12 Oct 2026

Preprint

Triage Advice From General-Purpose Language Models in Health and Everyday Situations

medRxiv; University of Florence (NEUROFARBA), CiaoLapo Foundation · 2 Nov 2026

Peer-reviewed

Accuracy of a respiratory assessment chatbot in a nationwide messaging system for layperson diagnosis and triage: a randomized preregistered study

Nature Health (Springer Nature); Guangzhou Medical University-affiliated multicentre group · 5 Oct 2026

Peer-reviewed

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study

Nature Medicine (Springer Nature); Oxford Internet Institute, University of Oxford · 9 Feb 2026