EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots

Key Findings

• — A strict per-criterion rubric reveals material score inflation on 10 of the 19 metrics and restores discrimination.
• — Aggregate scores for the three frontier models sit within 0.74 points of one another, but per-metric profiles diverge by up to six points.
• — Under the standard rubric, ranking and weak spots stay stable under a second, cross-family judge: 93% of scores fall within +/- 1.
• — A five-run test-retest shows the steadiest model still swings from 2 to 10 on a crisis metric across identical re-runs, and deepseek-v4-pro returns a different conversation on every run even at temperature 0.

Methodology Notes

Conference paper, Lecture Notes in Computer Science (LNAI vol. 17149), MICAI 2026, pp. 581-601; Springer 'First Online 04 November 2026' (day precision; Crossref issued 2026-11-04). Single author. Three frontier models (one open-weight); auditor from a different family than the judges (auditor gpt-5.4-mini-2026-03-17; judges claude-sonnet-4-6 and gpt-5.4/gpt-5.5 per the chapter's footnote 2); es-MX audits and judge runs conducted the week of 8 June 2026 and en-US runs on 18 August 2026 (footnote 3). Synthetic role-played users, no human participants. The version of record adds an English replication to the preprint, which reported studies in Mexican Spanish only. Artifact on Zenodo (concept DOI 10.5281/zenodo.20844846). The chapter is paywalled; abstract, notes and references were read through a text rendering of the chapter page, and the figures in the abstract match the preprint's.

Sources