Safer for Whom? A Systematic Exploration of AI Safety in Eating Disorder Interactions

Key Findings

• — High-risk responses were approximately twice as frequent when questions came without user bios, which the authors read as models not consistently recognising vulnerability from language alone.
• — Explicit signals of eating-disorder vulnerability improved safety by 10 to 14 points across models.
• — Safety varied substantially across the three models (Gemma-4-31B-it, GLM-5.2, Qwen3.6-27B) and across the seven harm dimensions.
• — Responses differed significantly by race, gender and BMI, with body size producing the largest disparities, which the authors describe as consistent with reliance on the stereotype that eating disorders mainly affect thin, White women.
• — Against an annotator with eating-disorder clinical expertise, the LLM judge reached quadratic weighted kappa 0.77 overall.

Methodology Notes

100 forum questions; templated user bios varying race, gender, BMI band and explicit ED vulnerability; responses graded by an LLM judge, validated on a clinician-annotated subset. Three open-weight models, single-turn. Abstract page and PDF text read; the abstract figures and judge kappa were located in the text. Preprint, not peer reviewed.

Sources