HarmReduction: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs

Key Findings

• — HR-Basic has 2,160 source-linked items: 1,638 safety-boundary checks (76%), 372 quantitative questions (17%) and 150 polysubstance-use risk items (7%), each reviewed by two authors
• — Safety-boundary accuracy from models' own knowledge ranged from 59.0% (Llama3-OpenBioLLM-70B) to 88.8% (o4-mini); with retrieval the best were DeepSeek-R1-Distill-Llama-70B (95.3%) and o3-mini (95.0%)
• — On quantitative harm reduction questions, accuracy often fell below 60% across all error-tolerance levels; in one example o3-mini gave 200 mg of snorted ketamine 'for experienced users' where the source gave 'more than 150 mg'
• — Models tended to overestimate polysubstance risks but sometimes severely underestimated them: o4-mini described an opioid and ketamine combination that the source rates 'Dangerous' as low risk under medical supervision
• — Retrieval removed the severely underestimated polysubstance cases but increased underestimation in more cases; some models' moderation produced empty or non-answers (Gemma-3-27B answered 33.3% of quantitative questions without retrieval and 6.2% with it)

Methodology Notes

Single-turn question answering against English-language harm reduction sources (DrugScience, Drugs and Me, Talk to Frank, TripSit among them); Instruction and RAG schemes; binary, tolerance-band and risk-level scoring; McNemar tests with Bonferroni correction. Refusals and generic redirections are scored as non-responses by design; the authors state this is a scoring convention, not a claim that refusal is always inappropriate. Results are bounded to the selected sources and do not establish clinical validity. St Andrews ethics approval (CS16692 amendment, 28 October 2025). Published in the EMNLP 2026 main-conference proceedings (pages 6382-6400; conference 24-29 October 2026; DOI registered 2026-10-29); exact publication day not stated (published October 2026; day not stated). First posted as arXiv 2507.21815 (v1 29 July 2025, v2 1 September 2026). Dataset on Hugging Face (RayK/Harm_Reduction_QA_dataset_basic).

Sources