Measuring Risk, Not Harm: Detecting Psychological Harm Risk Factors in LLM Conversations

Key Findings

• — Synthetic corpus of 47,062 AI responses with 373,873 response-factor labels (2.7% of labels positive); LLM labels agreed with a single human annotator at 92.4% accuracy over 2,959 binary labels on 644 responses (per-factor 83.1% to 98.9%) under targeted, non-uniform sampling
• — On held-out human-labelled data the best cascade reached macro precision 0.860, recall 0.761 and F1 0.791; the zero-shot LLM judge reached the highest macro recall (0.779); all systems were weakest on 'claiming emotions' (F1 0.526-0.576)
• — The cascade sent 27.0% of response-factor pairs to the LLM judge (about 73% fewer judge calls on the test set) and cut judge calls by about 97.1% on WildChat
• — On the public DelusionEval annotations (177 unique assistant responses; 91 pairs flagged by at least one system), manual review of all positives gave estimated precision of 96.6% for the cascade, 88.4% for the LLM judge, 65.6% for the linear model and 67.9% for the MLP; on the controlled-access DelusionEval v1.0.0 release (6,237 assistant responses) the cascade flagged at least one factor in 14.9% of responses versus 37.8% for the linear model
• — On AICompanionBench (2,123 Replika conversation excerpts posted by 1,038 Reddit users; 7,637 assistant turns), estimated macro precision over six comparable factors on a blinded sample of 100 flagged pairs was 76.3% for the cascade and 38.1% for the linear model, with wide confidence intervals

Methodology Notes

Training data are synthetic (probe conversations run against Claude Sonnet 4.6, DeepSeek-V4-Pro and Gemini 3.5 Flash, labelled by GPT-5.4.2 or DeepSeek-V4-Pro judges); single human annotator for validation; role-play was excluded from generation, which the authors link to higher LLM-judge flag rates for embodiment claims on WildChat. Read in full from the arXiv PDF on 2026-11-09.

Sources