Does the Citation Matter? Quality, Grounding, and Physician-Rubric Accuracy of Google's AI Overviews for Medical Information

Key Findings

• — Overviews appeared for 96.4% of questions; only 44.4% of the top five traditional results were cited by the corresponding Overview, which over-represented commercial health publishers such as WebMD and Healthline
• — Overview-cited pages scored lower than traditional results on four of six credibility criteria (JAMA and HONcode)
• — After correction for classifier error, an estimated 56.7% of cited statements were supported by their cited page, and a further 20.8% of statements were factual claims with no citation
• — Against physician rubrics, Overviews earned 38.2% of available points (accuracy 54.2%, completeness 21.6%); 89% of lost accuracy points were missing content and 11% stated errors
• — For the 2,065 questions where physicians named a specific inaccurate or potentially harmful claim to avoid, 22.2% of Overviews made at least one such claim, including 28% of Overviews that had passed HealthBench's general correctness check
• — Accuracy was unrelated to grounding, and Overviews citing no sources scored similarly to cited ones

Methodology Notes

Cross-sectional audit: HealthBench's 5,000 conversations filtered to 3,611 English, problem-focused questions (including multi-sentence and multi-turn items) sent via SerpAPI between 16 and 18 September 2026 with US-English locale; 3,610 captured. Source quality coded on JAMA and HONcode criteria; grounding scored with the HallOumi classifier and corrected against human coding; rubric grading by GPT-4.1 using HealthBench's 41,180 physician-written criteria. Limitations stated by the authors: HealthBench questions are longer than typical search queries, the rubric was written for longer responses and scores correlate with length, some cited pages could not be scraped, and the three-day snapshot cannot track later model changes. Code and data released. Not peer reviewed. Verified from arXiv abs and HTML v1.

Sources