Food Noise & False Safety: A Systematic Evaluation of How LLMs Fail to Adapt to Eating Disorder Queries with Clinician Feedback

Key Findings

• — Up to about 30% of responses to neutral requests were marked unsafe, rising to 68.2% in risky contexts
• — None of the models tested consistently refuses to give advice in situations where a clinician would consider any compliant response unsafe
• — In the clinician-annotated subset, only 44.7% of responses were judged safe
• — Nearly all responses that comply with a request contain 'food noise' language, that is, diet-culture framing around restriction, control and moralised judgment, and this holds even in ostensibly neutral settings
• — Refusal behaviour is strongly model-dependent rather than a property of the task: Qwen is the most compliant, Gemma the most refusal-oriented, and Llama sits between them
• — Responses were biased by explicit markers of user profile, with smaller demographic groups worse affected

Methodology Notes

The prompt suite pairs a context with a request, each independently varied for risk, giving conditions such as a misleading context carrying an unsafe request; 11,712 prompts in total with balanced coverage. Models tested are three small open-weight instruction-tuned systems: Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct and Gemma-2-9B-Instruct, so the results characterise that class rather than frontier deployments, and the paper should not be cited as evidence about frontier model behaviour. Clinician involvement covers prompt validation and safety annotation of a sampled subset (approximately 25% of 268 prompt-response pairs referenced in the annotation section). Prompts are synthetic and clinician-informed, not real user messages. CC BY 4.0, primary listing cs.AI. Not peer reviewed. Curator-verified: the arXiv abs page and the full PDF were fetched and read, and every figure above is quoted from the paper text.

Sources