Temporal and cross-site validation of an AI system for self-harm detection

Key Findings

• — The system combines extensive text normalisation with a self-harm classifier using 1,931 selected features.
• — Test-set performance was PR AUC 0.84 (95% CI 0.82 to 0.86).
• — Prospective validation on 329,655 triage notes from the same hospital over the following four years held at PR AUC 0.84 (95% CI 0.83 to 0.85).
• — External validation on 316,877 notes from a regional hospital 150 km outside Melbourne fell to PR AUC 0.78 (95% CI 0.77 to 0.79).
• — Text normalisation transferred; the degradation came from linguistic domain shift and a different presentation mix, with more medication ingestion in the regional setting.
• — Total validation corpus 646,532 manually annotated free-text triage notes across two Australian hospitals, 2012 to 2021.

Methodology Notes

Free-text emergency-department triage notes manually annotated for self-harm at two Australian hospitals; PR AUC is the reported metric, which is the appropriate choice for a rare positive class. Clinical notes, not conversation, so the transferability result applies to the linguistic-domain-shift mechanism rather than directly to chat data. Published 2026-09-11 in PLOS Digital Health, CC BY 4.0, DOI 10.1371/journal.pdig.0001667; abstract and metadata verified at Crossref.

Sources