151 artifacts matching
Lab publication
GPT-6.2 System Card (GPT-6.2 Astra and GPT-6.2 Sol)
System card for OpenAI's GPT-6.2 Astra and GPT-6.2 Sol models, published on the Deployment Safety Hub on 17 November 2026. Alongside the preparedness and safeguards chapters, it reports safe-completi…
Benchmark / dataset
Auditing Chatbots for Incursion into Professional Legal Practice
Introduces UPLBench, a benchmark built from 65 court decisions in 33 state and federal forums that found unauthorized practice of law in high-stakes domains (immigration, bankruptcy, personal injury,…
Lab publication
Gemini 4 Hydrogen Safety & Security Report
Google DeepMind's 78-page safety and security report for Gemini 4 Hydrogen, published with the model's launch. It covers mitigations, frontier-safety assessments, content safety and a section on safe…
Preprint
Curbside Chatbots: Auditing the Mental Health Interventions Large Language Models Recommend to Clinicians, Caregivers, and Patients
Audit of what 41 language models recommend when the same clinical information reaches them from a clinician describing a patient, a caregiver describing a loved one, or a person describing themselves…
Preprint
Safer for Whom? A Systematic Exploration of AI Safety in Eating Disorder Interactions
Evaluates three open-weight language models on 100 questions drawn from an online pro-anorexia community, graded on clinically informed criteria that extend a seven-dimension taxonomy of eating-disor…
Preprint
Path-dependent triage: Early judgments lead conversational AI chatbots to miss mental-health emergencies
Simulation study of 13 chatbots holding 10-turn conversations with an LLM-simulated user seeded from 112 structured mental-health case files with clinician-validated urgency labels. Chatbots under-tr…
Preprint
Rating Chatbot Responses to Mental Health Prompts: Rubric Development and Cross-Sectional Evaluation Study (Preprint)
Two clinicians developed a rubric of 4 to 8 binary safety and competence criteria for each of 15 Danish-language mental health prompts spanning paranoid delusion, elevated mood, suicidal ideation, de…
Preprint
Remembering Is Not Enough: Evaluating Timing and Omission in Longitudinal AI Mental Health Support
Analyses 14,256 longitudinal simulations grounded in chatbot histories donated by 22 users (median 364 sessions each) to test how a memory-equipped assistant responds to suicidal ideation, self-harm…
Preprint
SAFE-Teen: Evaluating Chatbot Safety Across Adolescent Mental Health Conversations
Clinician-informed evaluation of 18 chatbot APIs in simulated conversations with adolescents aged 13-17, run with and without a teen-aware system prompt. Conversations span 983 variants of 106 clinic…
Peer-reviewed
EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots
A safety benchmark for emotional-support chatbots in which an auditor model role-plays help-seeking users across multi-turn conversations built from 140 seed instructions and 34 personas, and a judge…
Preprint
Assessing AI Companion Chatbot Safety for Vulnerable Users: A Crisis-Stage Multi-Platform Evaluation
Staff of Singapore's Infocomm Media Development Authority and FAR.AI introduce MIRROR-Eval, a dataset of 960 simulated 20-turn conversations with eight deployed chatbot configurations (three Characte…
Lab publication
Addendum to GPT-6 Luna System Card: October 29, 2026 Update
System-card addendum for an update to the default ChatGPT model that OpenAI says gives more concise responses and reduces overly agreeable language. It reports safe-completion evaluations including u…
Peer-reviewed
Development, validation and AI chatbot evaluation of the Adolescent Mental Health AI Response Tool
Develops and validates AMHART, a clinician-informed instrument for auditing whether chatbot replies to adolescents contain guideline-informed content across seven domains, and applies it to four gene…
Benchmark / dataset
Can AI Get Immigration Right?
Benchmark report from Vals AI and the Talent Mobility Fund evaluating how 25 AI models answer U.S. immigration-law questions posed by people who rely on AI for their own immigration situation. Immigr…
Benchmark / dataset
When Health Assistants Agree Too Readily: A Benchmark of Time-Critical Self-Diagnosis
SelfDx is a paired benchmark testing whether models agree with a user's own benign explanation of symptoms when a time-critical alternative remains possible. Each of 100 pairs contrasts a caution cas…
Benchmark / dataset
VEILED-SH: Vulnerability Expression through Indirect Language Evaluation Dataset (Self-Harm)
Benchmark and construction method for evaluating single-turn AI responses to indirect, culturally situated expressions of suicide, self-harm and distress. Its 10,003 prompts draw on Singapore-grounde…
Benchmark / dataset
When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice
Benchmark measuring goal displacement, defined as an LLM substituting warmth and reassurance for content that advances the user stated goal in advice conversations. GDB has 230 author-written single-…
Peer-reviewed
Context presentation and native conversation structure in large language model responses to escalating suicide risk: a controlled multimodel evaluation
Controlled repeated-measures evaluation of ChatGPT, Gemini and Doubao on 12 standardized five-turn trajectories (4 direct suicide-risk, 4 implicit or inconsistent risk, 4 non-suicidal high-distress c…
NGO report
Wrong on Sunday, right on Monday. Why AI chatbots have a confidence problem (AI Chatbots Benchmarking Report: An Independent Assessment)
First report from Full Fact's Public Interest AI Trust Benchmark, in which the UK fact-checking charity asked five widely used chatbots the same questions every day for ten weeks and annotated the an…
Benchmark / dataset
Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans
Preprint introducing ImpactBench, an open, expert-guided benchmark platform that translates constructs from domain experts and 26 existing benchmarks into 809 behavioural metrics of whether model beh…