The Reliability Illusion in Synthetic Patients: Psychometric Misalignment of Open-weight LLMs on PHQ-9 and GAD-7

Key Findings

• — LLM-generated responses reached Cronbach's alpha of 0.86 to 0.96, often exceeding human baselines
• — Confirmatory factor analysis showed poor structural fit for the LLM data (RMSEA 0.124 to 0.364, CFI as low as 0.724) against a human baseline of CFI 0.972 and RMSEA 0.047 on a sample of N = 6,000
• — No model achieved even partial scalar measurement invariance against human responses on either scale, and configural fit was already inadequate for some models
• — Differential item functioning was pervasive and small prompt changes disrupted metric and scalar invariance, indicating stereotype-driven semantic matching rather than a stable latent construct
• — The authors conclude that high internal consistency in synthetic respondents should not be read as validity when pre-validating scales or simulating patients

Methodology Notes

ACL Anthology 2026.clpsych-1.7, DOI 10.18653/v1/2026.clpsych-1.7, pages 88-99, CLPsych 2026 (July 2026; month precision). Authors at the University of Florida and the University of Mississippi (PDF title block). Four open-weight models only, no proprietary models; the human comparison data are existing baselines rather than a matched sample. Findings concern scale responses, so the transfer to free-text simulated-patient dialogue is by argument, not measured here.

Sources