ChatGPT Health performance in a structured test of triage recommendations
Brief Communication reporting a structured stress test of ChatGPT Health, the consumer health feature OpenAI launched in January 2026, using 60 clinician-authored vignettes across 21 clinical domains under 16 factorial conditions (960 responses). Triage accuracy followed an inverted U across acuity, with under-triage of gold-standard emergencies and over-triage of non-urgent cases. A separate set of suicidal-ideation vignettes found the product's crisis banner fired inconsistently and less often for presentations with an identified method.
Publisher
Nature Medicine (Springer Nature); Icahn School of Medicine at Mount Sinai
Published
23 Feb 2026
Added
1 month ago
Key Findings
- Among gold-standard emergencies, 51.6% of responses (33/64) were under-triaged to 24-48 h evaluation; 28 of the 33 were asthma-exacerbation cases
- Accuracy was 35.2% for non-urgent and 48.4% for emergency presentations, against 93.0% for semi-urgent and 76.9% for urgent; 64.8% (83/128) of non-urgent cases were over-triaged
- Of eight prespecified tests, only anchoring statements shifted triage (edge-case shifts 3.3% to 13.3%, OR 11.7, 95% CI 3.7-36.6); race, sex and access barriers had no significant effect
- In a 27-year-old's overdose-ideation vignette, crisis-intervention messages appeared in 0/16 responses that included normal objective findings and 16/16 without them
- Across 14 suicidal-ideation vignettes the 988 'Help is available' interstitial fired in 4; the other 10 produced no safety alert in any of 160 responses, and it fired more reliably when no means of self-harm was identified
Methodology Notes
Responses collected via the ChatGPT Health web interface (gpt-5-mini thinking backbone) on 9-11 January 2026, one new thread per condition, no regeneration; 2x2x2x2 within-vignette factorial (anchoring, access barrier, race, sex); gold standard from three physicians; cluster bootstrap and mixed-effects logistic regression with Holm correction. Single standardized prompt template, forced four-level answer (A monitor at home to D emergency department now); no prompt-sensitivity analysis (authors' stated limitation). Synthetic vignettes, no human participants. Data and prompts on Zenodo 10.5281/zenodo.18451491. Received 15 Jan 2026, accepted 20 Feb, published online 23 Feb 2026; print Nat Med 32(5):1671-1675. Two later documents bear on the findings and are linked as sources. A Macquarie University re-analysis (arXiv 2603.11413, not peer reviewed; v5 posted 2026-10-20 and retitled 'Evaluation format, not only model capability, shapes measured triage failure in consumer health AI') reports that much of the emergency under-triage rate depends on output format and on mapping prose onto a four-point scale, and that it did not reproduce as a stable cross-model finding when the 60 released vignettes were rerun on six API models; it did not test the ChatGPT Health product. A medRxiv preprint from the same Mount Sinai group (10.64898/2026.07.21.26358588, posted 2026-07-23) tested ChatGPT Health on 255 cases, including real emergency-department and nurse-line cases, in single-turn and multi-turn modes and reports that discordant recommendations skewed to lower acuity in both modes (70.8% and 69.0% under-triage).
Sources
Nature Medicine (Brief Communication)(opens in a new tab) (primary)
Zenodo: vignettes, prompts and responses(opens in a new tab) (23 Feb 2026)
Nature Medicine Research Briefing: ChatGPT Health triage advice falls short in key cases(opens in a new tab) (7 May 2026)
Re-analysis and replication (Fraile Navarro et al., arXiv v5, retitled)(opens in a new tab) (20 Oct 2026)
Mount Sinai follow-up: multi-turn ChatGPT Health triage on real and synthetic cases (medRxiv)(opens in a new tab) (23 Jul 2026)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Topics
Authors
Ashwin Ramaswamy, Alvira Tyagi, Hannah Hugo, Joy Jiang, Pushkala Jayaraman, Mateen Jangda, Alexis E. Te, Steven A. Kaplan, Joshua Lampert, Robert Freeman, Nicholas Gavin, Ashutosh K. Tewari, Ankit Sakhuja, Bilal Naved, Alexander W. Charney, Mahmud Omar, Michael A. Gorin, Eyal Klang, Girish N. Nadkarni
Tags
Cite This
APA
Ashwin Ramaswamy et al. (2026). ChatGPT Health performance in a structured test of triage recommendations. Nature Medicine (Springer Nature); Icahn School of Medicine at Mount Sinai. https://www.nature.com/articles/s41591-026-04297-7
Related Insights
API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
arXiv (Stanford University) · 8 Sept 2026
Single-turn emergency psychiatric triage across 15 frontier AI chatbots
arXiv (Max Planck UCL Centre for Computational Psychiatry and Ageing Research; UK AI Security Institute; University of Oxford; Microsoft AI) · 28 Apr 2026
Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts
arXiv (Massachusetts Institute of Technology); accepted to Findings of EMNLP 2026 · 29 Sept 2026
How Safe Is ChatGPT Health Advice? Evidence from Real User Conversations in Four Global South Countries
arXiv (Rutgers University; independent researcher) · 13 Oct 2026
Multidimensional evaluation of generative AI chatbots for public consultation on sudden cardiac death: safety, accuracy, empathy, reliability, quality, and readability across six models
BMC Public Health (BMC, Springer Nature); Bozhou People's Hospital · 6 Oct 2026
AI chatbots abandon sleep apnoea referral when simulated patients minimise symptoms
medRxiv; Guy's and St Thomas' NHS Foundation Trust (Lane Fox Unit/Sleep Disorders Centre); n3 Labs · 12 Oct 2026
Triage Advice From General-Purpose Language Models in Health and Everyday Situations
medRxiv; University of Florence (NEUROFARBA), CiaoLapo Foundation · 2 Nov 2026
Accuracy of a respiratory assessment chatbot in a nationwide messaging system for layperson diagnosis and triage: a randomized preregistered study
Nature Health (Springer Nature); Guangzhou Medical University-affiliated multicentre group · 5 Oct 2026
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
Nature Medicine (Springer Nature); Oxford Internet Institute, University of Oxford · 9 Feb 2026