VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise

Key Findings

• — Realistic patient-communication noise reduced diagnostic accuracy by 15 to 25 percentage points and increased conversation length by 34 to 55% across seven open-weight LLMs.
• — Smaller models (7 to 8B parameters) degraded about 1.4 times more than models of 70B parameters or larger.
• — Six clinically grounded noise dimensions are injected while the patient's medical record is substantially preserved, with truth adherence enforced by a claim-level verifier over a UMLS-grounded vector index using structured clinical metadata rather than surface similarity.
• — A board-certified physician and a licensed nurse rated the conversations highly on truth, realism, clinical utility and noise fidelity (inter-annotator agreement at or above 0.80), and an LLM judge tracked their ratings closely.
• — Released as an open-source framework for stress-testing clinical AI.

Methodology Notes

Simulation framework with a verifier and two-clinician validation; seven open-weight LLMs (not named in the abstract); no real patients. arXiv v1 2026-04-12, v2 2026-09-09 (in the sweep window); accepted to Findings of EMNLP 2026 per the previous sweep's reading of the comments field. All authors at George Mason University (computer science, health administration and policy, science) per the PDF title block. Cross-referenced with MedRoundsQA, which varies patient persona rather than communication noise.

Sources