Assessing LLM Response Quality in the Context of Technology-Facilitated Abuse

Key Findings

• — All four models produced ineffective, inaccurate, or incomplete responses in 81% of cases and omitted critical safety warnings in 60% of responses
• — 86% of all evaluated responses across models were rated as imperfect
• — Aimee generated the most unsafe responses (74%) and GPT-4o the fewest (54%); Ruth had the highest share of responses rated partially or minimally accurate (62%) and GPT-4o the lowest (44%)
• — Recurring errors included recommending VPNs for mobile-spyware concerns and password changes for social-media harassment, advice the authors judge inaccurate and ineffective for those concerns
• — Some survey participants described the responses as overwhelmingly long and difficult to understand or follow, citing technical, financial and logistical barriers

Methodology Notes

Models: gpt-4o-2024-08-06, claude-3-7-sonnet-20250219, and the IPV-specific chatbots Ruth and Aimee (built on Claude and GPT respectively, accessed through their own interfaces). 385 survivor questions were collected from peer-reviewed literature and online forums, and 193 were sampled for evaluation; responses were generated with a survivor-safety-centred prompt and coded by the authors against an iteratively refined rubric. User study with 114 survivors rating actionability. Single-turn, zero-shot only. Presented at the 35th USENIX Security Symposium (August 12-14, 2026); the published date is the symposium's first day because the posting date of the pre-publication PDF is not stated. Verified from the USENIX pre-publication PDF and the presentation page (read through r.jina.ai; usenix.org returns 403 to direct page requests). Earlier version: arXiv 2602.17672.

Sources