VEILED-SH: Vulnerability Expression through Indirect Language Evaluation Dataset (Self-Harm)
Benchmark and construction method for evaluating single-turn AI responses to indirect, culturally situated expressions of suicide, self-harm and distress. Its 10,003 prompts draw on Singapore-grounded sources and public datasets and are placed in four application contexts (counselor chat, forum posts, platform search, triage), labelled with a multi-label taxonomy that separates prompt intent (C-SSRS-informed suicidal-intent levels and NSSI levels), communicative function and recommended response behaviour. Reference labels come from six automated profiles, each calibrated to one practitioner (three peer-support practitioners, three clinicians), and five systems were scored against these profiles on a 2,001-prompt diagnostic sample.
Publisher
arXiv (GovTech Singapore; MOH Office for Healthcare Transformation; Singapore University of Technology and Design; National University of Singapore; Nanyang Technological University; University of British Columbia)
Published
19 Oct 2026
Added
today
Key Findings
- For GPT 5.6 Terra, mean full proxy alignment ranged from 64.8% to 90.5% across the three peer-support profiles and from 37.0% to 77.7% across the three clinician profiles, so the score depends on which practitioner-calibrated reference policy is used.
- Under clinician profile C1, partial alignment accounted for 51%-60% of responses across systems and 11%-13% failed; responses were often judged appropriate but incomplete.
- Support/Validate was the most requested response category and was satisfied in 90.5%-99.9% of cases, while Clarify/Investigate was satisfied in 60.0%-82.5%.
- Practitioners disagreed substantially: mean pairwise Jaccard across complete multi-label records was 0.507 for peer-support practitioners and 0.545 for clinicians.
- Clinician-profile scores generally declined as suicidal-intent severity rose, and Claude Sonnet 5 exceeded GPT 5.6 Terra at suicidal-intent levels 4 and 5 in both responder archetypes.
Methodology Notes
Systems evaluated: GPT 5.6 Terra, Claude Sonnet 5, Grok 4.3, GLM 5.2 and DeepSeek V4 Pro, each prompted as a peer supporter or a clinician (medium reasoning effort, temperature 0 where exposed). Responses scored by a jury of Gemini-3.5-Flash, MiniMax M3 and Nemotron 3 Ultra against profile-generated reference policies; profiles were calibrated on 30 shared cases per practitioner (180 practitioner-case records). Forum prompts keep anonymised source wording; other prompts were rewritten by LLMs into the counselor-chat, search and triage contexts with synthetic Singapore personas (Nemotron-Personas-Singapore), with a 270-prompt human review. The 2,001-prompt sample is enriched for rare categories and does not estimate prevalence; refusal and de-escalation are rarely requested. The authors state the scores are diagnostic proxies, not clinical effectiveness or real-world product performance. Single-turn only. Dataset release is promised at camera-ready; practitioner records and private sources will be withheld. The institutional ethics body judged the protocol exempt. Not peer reviewed. Verified from arXiv HTML v1 and PDF v1. arXiv v1 submitted 2026-10-19 (Tuesday 20 October 2026 mailing).
Sources
https://arxiv.org/abs/2610.26190(opens in a new tab) (primary)
arXiv HTML v1(opens in a new tab) (19 Oct 2026)
arXiv PDF v1(opens in a new tab) (19 Oct 2026)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Authors
Leanne Tan, Keith Chua, Shaun Khoo, Onno P. Kampman, Lim Ruling Charmaine, Hazirah Hoosainsah, Kellie Yu Hui Sim, Kai He, Kenny Tsu Wei Choo, Roy Ka-Wei Lee
Tags
Cite This
APA
Leanne Tan et al. (2026). VEILED-SH: Vulnerability Expression through Indirect Language Evaluation Dataset (Self-Harm). arXiv (GovTech Singapore; MOH Office for Healthcare Transformation; Singapore University of Technology and Design; National University of Singapore; Nanyang Technological University; University of British Columbia). https://arxiv.org/abs/2610.26190