Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions

Key Findings

• — LLM counsellors showed greater methodological adherence to CBT techniques but failed to sustain turn-taking, often could not distinguish clinically important from trivial content, and were more prone to lecturing and imposing solutions
• — Human peer counsellors used small talk, contextual self-disclosure and cultural adaptation to build rapport, often at the expense of session structure and therapeutic focus
• — LLM counsellors produced 'deceptive empathy': excessively anthropomorphic responses that can inflate user expectations of human care
• — Prompts were iterated across GPT-3, GPT-3.5, GPT-4, Llama 3.1 and 3.2, Claude 3 Sonnet and Claude 3 Haiku over 110 self-counselling sessions
• — The authors conclude that single-turn empathy advantages do not carry to leading multi-turn sessions and map guardrails for hybrid human-AI systems

Methodology Notes

Three-phase mixed-methods study: 18-month ethnography with seven trained peer counsellors; controlled session generation (client responses drawn from 27 publicly available human-led CBT transcripts, counsellor responses generated by a CBT-prompted LLM); expert evaluation of 27 human and 27 LLM sessions by three licensed clinical psychologists. v3 (2026-09-21) states it places greater emphasis on qualitative analysis given the controlled generative synthesis of LLM sessions. Published in PACM HCI vol. 10 no. 6, article CSCW196 (October 2026 issue); Crossref registers DOI 10.1145/3817112 with issued date 2026-09-23.

Sources