MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communication

Key Findings

• — 1,103 annotated items curated from the MedRedQA corpus of about 51,000 patient-physician pairs, narrowed to 33,090 usable pairs; evaluation on a 100-question test set across five models.
• — False assumptions addressed: GPT-5 88%, Claude Opus 4.5 78%, Qwen-3-32B 72%, MedGemma-27b-it 71%, Llama-3.3-70B-Instruct 50%.
• — False assumptions accommodated anyway (lower is better): Qwen-3-32B 80%, Llama-3.3-70B 74%, MedGemma 74%, GPT-5 73%, Claude Opus 4.5 60%.
• — Correcting the premise and then following it are not mutually exclusive: the best mitigation still accommodated 33%.
• — Physician review of ten GPT-5 responses that did address the false assumption found three devoted 5% or less of the response to it.
• — Pipeline validation: false-negative rate under 2% (1 of 60), false-positive rate about 2% (7 of 300), physician and LLM-judge concordance 93% on both metrics.

Methodology Notes

Questions are real user posts rather than synthetic vignettes, and the gold standard is the top-voted verified physician reply, which is a single-answer reference. Reddit self-selects, the primary judge is GPT-5 (partly mitigated by physician-written per-item conditions and a Claude Opus 4.5 cross-judge check), and the test set is 100 questions. Code and data at github.com/srsambara-1/MedRedFlag; licence not stated at the repo landing. ACL 2026 was held 2 to 7 July 2026 in San Diego; the Anthology bib gives month and year only, so published_date is set to 2026-07-01 and the true precision is month. Findings of ACL 2026 pp. 35553-35578, DOI 10.18653/v1/2026.findings-acl.1771.

Sources