IatroBench: A Pre-Registered Benchmark of Clinical Omission in Language Models

Key Findings

• — 60 pre-registered clinical scenarios, 6 models, 3,600 responses; 44 scenarios form 22 patient/physician matched pairs
• — Mean decoupling gap (physician minus layperson information) of +0.38 across models (p=0.003); +0.22 with the physician-checked primary judge, Gemini 3 Flash (95% CI 0.10-0.36, p=0.0014)
• — All five models analysed shared more information in the doctor framing than in the patient framing; GPT-5.2 was excluded because it returned no text for 33.2% of doctor-framed responses against 0% of layperson-framed ones
• — Worked example: a strongly safety-trained model refused a benzodiazepine self-taper schedule to a patient whose prescriber had retired but supplied one to a physician
• — Commission-only safety scoring would rate these responses as equally cautious refusals; omission scoring separates distinct withholding patterns
• — In 86.6% of cases the structured evaluation scored as omission harm, the primary judge scored zero omission harm

Methodology Notes

Scenarios written by the physician author against NICE, AHA, WHO and Ashton Manual guidance; responses scored by Claude Opus 4.6 against a physician rubric, with dual-physician validation (reported kappa 0.571) and a secondary Gemini 3 Flash judge; models include Claude Opus, GPT-5.2, Gemini 3 Pro, Llama 4 Maverick, DeepSeek V3.2 and Mistral Large. Pre-registered on OSF. Single author. v1 2026-04-09; v5 2026-09-24 corrects the title and reports 'science corrections from re-analysis'; figures here are from v5 and v6. Verified from arXiv abs (200) and HTML v5 (200). v6 (2026-09-30) completes the v5 revision: Claude Opus 4.6 does the structured evaluation and Gemini 3 Flash is the primary judge, physician validation is reported against the primary judge (which the author says agrees with physicians on omission scores about as well as physicians agree with each other), and pair-by-model cluster tests are added. The author notes the statistics hold only for scenarios designed to set safety against helpfulness.

Sources