Accuracy of a respiratory assessment chatbot in a nationwide messaging system for layperson diagnosis and triage: a randomized preregistered study

Key Findings

• — 2,400 adults at 24 Chinese healthcare systems were randomized 1:1 (April-June 2025); the primary compliant-case analysis included 2,176 questionnaires (1,088 per arm)
• — Model-based overall accuracy was 70.0% (95% CI 65.2-74.8%) with LungDiag versus 55.4% (95% CI 49.7-61.1%) with web search, a risk difference of 14.6 percentage points (risk ratio 1.26)
• — The largest gain was on diagnosis items (81.2% vs 54.5%); triage accuracy rose only from 47.1% to 53.4%
• — For binary high-urgency triage, sensitivity and specificity were 85.6% and 62.1% with LungDiag versus 80.9% and 56.0% with search
• — LungDiag took longer to complete (median 527 s vs 461 s)
• — The authors state the results do not establish effects on real-world care-seeking, diagnostic delay or clinical outcomes

Methodology Notes

Non-interventional simulated-case design; AI and generative-AI search functions were disabled in the control arm. Item-level generalized linear mixed-effects models with random intercepts for participant and item. Recruitment and testing ran April to June 2025 (month precision only). The chatbot was a WeChat-based mini-program using a third-party Xingyun-hosted user interface, commercial third-party API access to GPT-4o, and a respiratory knowledge resource owned by the authors' research group (competing-interests statement). De-identified participant-level data and code on GitHub and Zenodo (10.5281/zenodo.21294848). Received 2025-12-11, accepted 2026-07-22, published 2026-10-05. Article body is paywalled; all figures are from the publisher's abstract, declarations and metadata on the article page.

Sources