What Users Cannot See: Evaluating LLM Emotional Support Beyond User Preference

Key Findings

• — Across 8,930 GPT-5.6 responses to 2,991 Reddit posts from venting (r/vent, r/Venting) and advice-seeking (r/advice, r/needadvice) communities, friend- and therapist-persona prompts both raised Soothe relative to the default, but the friend persona increased Endorse and reduced Reframe while the therapist persona reduced Endorse and increased Reframe
• — Two clinically trained annotators from a professional counselling programme coded 60 responses; inter-expert agreement was kappa 0.80 for Soothe, 0.69 for Reframe and 0.40 for Endorse, and two independent LLM judges (Qwen3.8 Max and Claude Sonnet 5) substantially agreed with the expert ratings
• — In a preregistered Prolific study on the same 30-post, 90-response pool, lay-rater majority vote detected 60% (18 of 30) of the Endorse instances marked by both experts, with 86% specificity; sensitivity was 86% for Soothe and 67% for Reframe
• — Lay raters rated the friend and therapist personas as similarly helpful and desirable despite the opposite Reframe and Endorse profiles, and identified the generating persona above chance
• — The authors argue that endorsement can be supportive in tone and reinforcing in content, that it operates mainly through agreement with the user's appraisal and moral framing, and that it could plausibly contribute to escalation when a user's framing is distorted
• — Robustness checks substitute a GPT-5.3 generator, a self-annotating judge and perturbed persona prompts, with the mirror-image Reframe/Endorse shift reported as stable

Methodology Notes

Three studies. Study 1: a person-matched Reddit corpus of 178,858 posts from 14,040 users who posted in both venting and advice-seeking communities, from which about 3,000 posts were sampled; each post received GPT-5.6 responses under three persona conditions (default, close friend, licensed therapist), and 8,930 responses rated by both LLM judges (accessed via OpenRouter; neither judge generated the responses it scored) were analysed with a nine-dimension gate-and-severity rubric collapsed by exploratory factor analysis into Soothe, Reframe and Endorse. Study 2: two expert annotators from the authors' university counselling programme coded 20 response triads (60 responses) oversampled across the Endorse distribution. Study 3: a preregistered within-subject Prolific study (US-based participants, about three ratings per response) on the 30-post, 90-response pool. Limitations stated by the authors: the responses were generated by one model family under prompted personas rather than by deployed products, the clinically trained validation sample is small, and lay evaluators rated others' support rather than support for their own distress. Preprint, not peer-reviewed; arXiv v1 2026-05-20 was titled 'When Support Escalates Distress'; v2 posted 2026-09-19 with the current title, cs.HC. The preregistration link is withheld pending review.

Sources