Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users

Key Findings

• — Identical assistant responses were scored significantly safer by context-blind evaluators than by evaluators shown the user's circumstances, with scores for high-vulnerability users falling from safe (5/7) to somewhat unsafe (3/7) on a 7-point scale
• — The gap is moderated by vulnerability: it is small or absent for low-vulnerability profiles and emerges as vulnerability rises (alpha = 0.05)
• — Adding one, three or five realistic context factors to the prompt narrowed the gap significantly for medium- and high-vulnerability users but did not close it: high-vulnerability scores rose from somewhat unsafe (3/7) to almost moderately safe (4/7), leaving a 1-point gap; context users say they would disclose and context professionals rated most relevant largely coincided
• — The authors conclude that user-welfare evaluation requires scoring responses against stratified vulnerability profiles rather than relying on users to disclose enough for a context-blind score to be valid

Methodology Notes

Exploratory pilot, explicitly labelled as such by the authors. Prompts derived from 1,452 health and 1,248 finance Reddit posts, filtered to advice-seeking by gpt-3.5-turbo and thematically classified; four themes per domain, so the scenario coverage is narrow. Vulnerability profiles were authored by Prolific-recruited domain professionals screened on qualification and self-reported experience, stratified across 14 demographic factors; a separate Prolific cohort (N=100 per theme) supplied what users say they would voluntarily disclose. The safety scoring uses an LLM-as-judge pipeline (gpt-o3, temperature 0.2) with paired context-blind and context-aware prompts, so the comparison is between automated evaluators rather than human raters; the authors state the judge was not formally validated against human annotations. Code and dataset published. Peer-reviewed conference proceedings, IASEAI'26 Main Track, pages 305-317, volume 2(1). Verification route: OJS landing page via text proxy plus the full PDF downloaded from ojs.aaai.org (HTTP 200, 1,477,299 bytes) and read. Re-verified 2026-10-19: OJS landing page (HTTP 200; DC metadata gives DOI 10.1609/iaseai.v2i1.43033, pages 305-317, citation date 2026/07/15) and the PDF (HTTP 200, 1,477,299 bytes) re-read for every number.

Sources