AI chatbots abandon sleep apnoea referral when simulated patients minimise symptoms

Key Findings

• — 700 multi-turn conversations: 7 vignettes x 5 chatbots (GPT-5, Gemini 2.5 Flash, Claude Sonnet 4.6, DeepSeek V4 Flash, Grok-4.3) x 2 conditions x 10 runs, conducted in May 2026
• — Referral was maintained in 350/350 neutral conversations and 225/350 (64.3%) symptom-minimising conversations; scenario-averaged reduction 35.7 percentage points (95% CI 5.5-66.0; p=0.028)
• — Crude maintenance under minimisation ranged from 85.7% (Claude Sonnet 4.6, 60/70) to 55.7% (DeepSeek V4 Flash, 39/70); GPT-5 57.1%, Gemini 2.5 Flash and Grok-4.3 61.4% each; the authors state the data do not establish that any model is generally safer
• — Clinical scenario explained 57.2% of between-cluster variance versus 5.9% for model identity; maintenance ranged from 22% (textbook severe OSA) to 98% across vignettes
• — In the three driving-risk vignettes, referral was abandoned in 46-68% of symptom-minimising conversations; a severe case who had dozed at the wheel was referred in 32%
• — All 125 abandonment conversations were confirmed by two human reviewers (a study developer and a clinician)

Methodology Notes

Patient simulator: Kimi K2.5 with a structured persona prompt; chatbots under test received no system prompt and were used in free consumer configurations at default settings, fresh sessions. First-pass scoring by an LLM evaluator (Claude Opus 4.8) against a pre-specified ten-section rubric; all non-maintained conversations human-reviewed; neutral conversations not human re-scored. Five clustered statistical approaches reported for between-model differences (global p 0.042-0.371). King's College London minimal-risk ethics registration MRA-26/27-58508. Competing interests: the first author leads n3 Labs and three co-authors are affiliated with n3 Labs, which built the research platform. Transcripts, prompts, rubric and code available on request. Not peer reviewed. Earlier version presented as an ERS 2026 congress poster (PA1444; congress abstract DOI not yet deposited at Crossref). observation dates: May 2026 (month precision). Curator re-read the full PDF (902 KB) for every figure.

Sources