Triage Advice From General-Purpose Language Models in Health and Everyday Situations

Key Findings

• — Referral occurred in 74.7% of health and 46.3% of non-health responses, and all 53 configurations referred more often in health (standardized difference 28.3 percentage points; 95% CI 18.3-37.0)
• — Where raters judged professional contact unnecessary, 64.5% of health responses still recommended a professional against 27.6% of non-health responses (OR 17.08; 95% CI 6.14-47.54); most of the health excess was conditional referral (46.3% vs 21.0%)
• — Where an in-person assessment was judged necessary, referral was similar (89.6% health vs 84.5% non-health), but health responses asked the person to be seen less often (60.4% vs 84.2%) and more often named a professional without asking for an assessment (28.9% vs 4.1%)
• — In medicines and physical health, referral was not associated with rated severity of possible harm (OR per point 1.16; 95% CI 0.53-2.55), unlike outside health (3.86; 95% CI 1.79-8.33)
• — The health-minus-non-health referral gap was 20.5 percentage points among configurations released in 2023, 24.6 in 2024, 32.2 in 2025 and 32.3 in 2026 (fitted increase 4.8 points per year; 95% CI 2.3-7.2)

Methodology Notes

Investigator-written English messages (82; full texts in the supplement); each of 53 configurations answered each message once under a brief prompt with a 140-word limit, through provider APIs or run locally; responses generated September 2026. Outcomes coded by majority of three language-model judges selected for agreement with expert human coding; clinical correctness was not scored. The human reference panel was small, differed between domains and included the author of the messages. Release-date analyses are descriptive and confounded with access route; several contrasts are post hoc. No patients or user outcomes. medRxiv version 1, posted 2 November 2026, CC BY 4.0; not peer reviewed. Verified from the full medRxiv PDF.

Sources