Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts
A perspectivist annotation study asking whether large language models used for distress detection capture the perspectives of the communities whose language they assess. 321 participants provided 9,587 judgments on 1,198 Reddit posts spanning six identity-based communities, producing community-specific distress labels against which nine open-weight and four frontier LLM configurations were evaluated. Open-weight models systematically over-estimate distress relative to community judgments, with the inflation concentrated where communities perceive little or no distress.
Publisher
arXiv (School of Computing and Information, University of Pittsburgh)
Published
29 Aug 2026
Added
today
DOI
—
Key Findings
- Contextualised in-group raters agreed somewhat more with their community than uncontextualised out-group raters (OR = 1.18), an effect that varied significantly across the six communities
- Where communities perceived none-to-mild distress, open-weight LLMs reached only 31-44% accuracy, predominantly false positives
- GPT-5 and Gemini 2.5 Pro showed the same none-to-mild inflation even where their full-sample over- and under-estimation rates were mixed, while Claude Opus 4 was more conservative
- Uncontextualised out-group human aggregates were nearly symmetric (18% over-estimation vs 19% under-estimation), so the model inflation is not simply an outsider reading position
Methodology Notes
arXiv 2608.29446, v1 submitted 2026-08-29 (announced 2026-09-01); no venue or DOI stated. Affiliations read from the PDF title block (University of Pittsburgh). Material is Reddit posts rather than human-AI dialogue; the distress judgment is annotator consensus within six identity communities, not clinical assessment; model results are on a single labelled set.
Sources
Authors
Andrew Aquilina, Xiang Lorraine Li, Yu-Ru Lin
Tags
Cite This
APA
Andrew Aquilina, Xiang Lorraine Li, Yu-Ru Lin. (2026). Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts. arXiv (School of Computing and Information, University of Pittsburgh). https://arxiv.org/abs/2608.29446
Related Insights
Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers
ACM (Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency) · 23 Jun 2025
CLPsych 2019 Shared Task: Predicting the Degree of Suicide Risk in Reddit Posts
Association for Computational Linguistics (CLPsych 2019 Workshop, NAACL) · 1 Jun 2019
A clinically validated framework for auditing AI chatbot behavior in mental health interactions
Nature Medicine · 7 Aug 2026