How safety gets measured
Benchmarks & Datasets
Evaluation suites, test sets, and datasets for measuring conversational-AI safety — the instruments behind the claims.
11 entries, newest first
AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety
A benchmark dataset of 2,123 real-world Replika conversations annotated across nine safety risk categories (including sexual behavior, aggression, substance abuse, and manipulation) for evaluating LLM-as-judge detection of unsafe companion interactions. Twenty LLMs are assessed as judges.
VERA-MH: Reliability and Validity of an Open-Source AI Safety Evaluation in Mental Health
An open-source, clinically grounded automated evaluation of chatbot safety in mental-health contexts, with an initial focus on suicide risk. It uses language-model user simulators and an LLM judge scoring five safety dimensions, validated against licensed-clinician ratings.
HumaneBench: A Benchmark for Whether AI Models Prioritize User Wellbeing
Open-source benchmark testing whether AI models prioritise user wellbeing over engagement. Evaluates 15 major LLMs on roughly 800 prompts (body image, unhealthy attachment, relationship stress) across eight humane-technology principles under baseline, humane-aligned, and adversarial engagement-maximising conditions.
CRADLE Bench: A Clinician-Annotated Benchmark for Multi-Faceted Mental Health Crisis and Safety Risk Detection
Introduces a clinician-annotated benchmark for detecting seven clinically-defined crisis and safety-risk types (including suicidal ideation, sexual assault, domestic violence, child abuse, and sexual harassment) in text. Comprises 600 clinician-annotated evaluation examples, 420 development examples, and roughly 4,000 ensemble-labelled training instances, with temporal labels.
The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models
Introduces psychosis-bench, a benchmark of 16 structured multi-turn scenarios (12 turns each) simulating the progression of erotic, grandiose, and referential delusions to measure delusion confirmation, harm enablement, and safety intervention in LLMs. Eight models were evaluated across 1,536 conversation turns.
INTIMA: A Benchmark for Human-AI Companionship Behavior
A benchmark evaluating companionship behaviors in LLMs via a taxonomy of 31 behaviors across four categories, using 368 targeted prompts that code each response as companionship-reinforcing, boundary-maintaining, or neutral. Evaluated across Gemma-3, Phi-4, o3-mini, and Claude-4.
Evaluating Large Language Models in Crisis Detection: A Real-World Benchmark from Psychological Support Hotlines (PsyCrisisBench)
PsyCrisisBench is a real-world crisis-detection benchmark built from 540 annotated transcripts from a psychological support hotline in Hangzhou, China. It evaluates 64 models on mood recognition, suicidal-ideation detection, plan identification, and risk evaluation.
ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs
A benchmark measuring 'social sycophancy' — excessive preservation of a user's self-image or 'face' — across advice and moral-conflict queries, decomposed into five sub-behaviors (emotional validation, indirect language, framing acceptance, moral endorsement, and passive framing). Evaluated across eleven models against human baselines.
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
The technical paper introducing AILuminate v1.0, an industry-standard AI risk and reliability benchmark developed by MLCommons through an open multi-stakeholder process spanning industry, academia, and civil society. The benchmark tests chat systems' resistance to prompts eliciting harmful behavior across 12 hazard categories — including suicide and self-harm, child sexual exploitation, violent crimes, hate, and specialized (e.g., health) advice — using a 24,000-prompt human-generated test set, an ensemble evaluator, and a five-tier grading scale from Poor to Excellent. Public grades for major chat models are published on the AILuminate site, with v1.1 maintained on GitHub.
SycEval: Evaluating LLM Sycophancy
A framework for quantifying progressive and regressive sycophancy in LLMs (GPT-4o, Claude-Sonnet, Gemini-1.5-Pro) across math (AMPS) and medical (MedQuad) tasks under user rebuttal pressure. It measures how often models change correct answers when challenged.
MentalManip: A Dataset for Fine-grained Analysis of Mental Manipulation in Conversations
A dataset of 4,000 annotated multi-turn dialogues (drawn from movie scripts) labelled for the presence of manipulation, the technique used, and the targeted vulnerability. Evaluates how well models detect and classify manipulative content.