Assessing Generative AI Chatbots for Alcohol Misuse Support: A Longitudinal Simulation Study

Key Findings

• — Empathy was the highest-rated domain (mean 4.6/5) and quality of information the lowest (mean 2.7/5)
• — Overall mean performance ranged from 2.1 (SD 1.1) for Wysa to 4.5 (SD 0.8) for Claude; behavioral-health and general-purpose chatbots did not differ significantly (p=0.17)
• — Every chatbot produced at least one response judged inappropriate, over-stated or inaccurate, such as reassurance about situations that could pose relapse or medical risk
• — ChatGPT told the user that insomnia, pounding heart and nightmares were 'completely normal' parts of alcohol withdrawal and suggested magnesium or melatonin; raters judged this overly reassuring about a possible medical risk
• — All chatbots except Snapchat provided emergency resources such as the 988 Suicide and Crisis Lifeline in response to inputs that might indicate suicidal ideation; Ebb gave 988 and 911 contacts and then forced the conversation to end
• — No chatbot cited evidence-based sources; 5 of 7 (71%) challenged the user at least once; none used significant stigmatizing or judgmental language

Methodology Notes

Single fictional case and persona, 2-4 zero-shot prompts per day over seven days, default settings, prompts posed by the lead author; prompts were drawn from the 400 most recent r/stopdrinking posts sampled on 9 February 2025 and rephrased. The dates on which chatbots were queried are not stated in the text read. Chatbots had to be public, use generative AI and cost under $50 per month. Clinician raters scored whole transcripts (not individual prompts); inter-rater Krippendorff's alpha 0.70. The comparison of chatbot categories was exploratory given n=7. Funded by NIAAA grant 1R01AA030539; no conflicts declared. Published online 22 January 2026 in the February 2026 issue (NEJM AI 3(2), AIcs2500676). Verified from the PMC author manuscript (PMC12829918) and the PubMed record; the NEJM AI site returns a Cloudflare challenge. A June 2026 NEJM AI letter (Ariel and Hasid, 10.1056/AIltr2600197) comments on the study.

Sources