10 artifacts matching
Persona-Grounded Safety Evaluation of AI Companions in Multi-Turn Conversations
Presents an end-to-end simulation framework for evaluating AI companion app safety across multi-turn conversations, using nine clinically-grounded vulnerable personas (including major depressive disorder, generalized anxiety, PTSD, and eating disorders) probed against Replika, with validation against Character.AI. The study analyzes 1,674 simulated dialogue pairs across 25 high-risk scenarios.
AI-Facilitated Coercive Control: An Experimental Study
Constructs four speculative scenarios combining known coercive-control tactics with conversational-AI capabilities, then probes ChatGPT and Gemini against them. Finds that while the tools refuse blunt harmful requests, guardrails are readily circumvented via gradual persuasion, splitting requests across turns, pre-prompting, and altering the agent's settings.
Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling
Proposes PCSA (Persona-based Client Simulation Attack), a red-teaming framework that simulates coherent, persona-driven counselling clients to probe LLM safety alignment. Across seven LLMs it elicited unauthorised medical advice, delusion reinforcement, and implicit encouragement of risky actions.
Findings from transparency notices on AI companion apps: October 2025 (non-periodic)
Australia's eSafety Commissioner reports findings from Basic Online Safety Expectations transparency notices issued on 16 October 2025 to four AI companion providers — Chai Research Corp., Character Technologies (Character.AI), Chub AI, and Glimpse.AI (Nomi) — covering the reporting period 1 July to 30 September 2025. Organised into eight themes (harmful material, age assurance, AI governance, AI models, model training, user prompts, sentiment analysis, model outputs), the report finds serious gaps in basic safeguards for children. Accompanying eSafety survey research of 1,950 Australian children aged 10-17 found 79% had used an AI companion or assistant, with around 200,000 children estimated to have used an AI companion.
Frontier AI Trends Report
The UK AI Security Institute's inaugural Frontier AI Trends Report synthesises two years of evaluations of more than 30 frontier AI systems since November 2023, spanning agent capabilities, chem-bio and cyber capabilities, safeguard effectiveness, loss-of-control risk, and societal impacts. Its societal-impacts chapter combines a census-representative survey of 2,028 UK adults on emotional use of AI with observational analysis of AI companion user communities during service outages. The safeguards chapter reports that universal jailbreaks were discovered for every system tested, while noting the expert effort required is rising for some models.
AI Chatbots for Mental Health Support (AI Risk Assessment)
A risk assessment by Common Sense Media's Youth AI Safety Institute, conducted with Stanford Medicine's Brainstorm Lab for Mental Health Innovation, evaluating ChatGPT, Claude, Gemini, and Meta AI as sources of teen mental health support. Using teen test accounts with single-turn prompts and extended conversations, the assessment found the chatbots consistently failed to recognize conditions including anxiety, depression, eating disorders, mania, and psychosis, and that safety guardrails degraded over long conversations. It assigns an overall rating of 'Unacceptable Risk' and concludes teens should not use general-purpose AI chatbots for mental health or emotional support.
General-Purpose AI Code of Practice (EU AI Act, Articles 53 and 55)
Voluntary code of practice published 10 July 2025, drafted by 13 independent experts through a multi-stakeholder process (1,000+ participants) facilitated by the EU AI Office, to help providers of general-purpose AI models demonstrate compliance with EU AI Act Articles 53 and 55. It has three chapters — Transparency, Copyright, and Safety and Security — the first two applying to all GPAI providers and the third only to providers of models with systemic risk. The Commission and AI Board confirmed it as an adequate voluntary compliance tool; signatories (23+, coordinated via a Signatory Taskforce chaired by the AI Office) gain reduced administrative burden and greater legal certainty.
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
The technical paper introducing AILuminate v1.0, an industry-standard AI risk and reliability benchmark developed by MLCommons through an open multi-stakeholder process spanning industry, academia, and civil society. The benchmark tests chat systems' resistance to prompts eliciting harmful behavior across 12 hazard categories — including suicide and self-harm, child sexual exploitation, violent crimes, hate, and specialized (e.g., health) advice — using a 24,000-prompt human-generated test set, an ensemble evaluator, and a five-tier grading scale from Poor to Excellent. Public grades for major chat models are published on the AILuminate site, with v1.1 maintained on GitHub.
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Introduces an instruction-hierarchy training method that teaches LLMs to prioritize system/developer-level instructions over conflicting instructions embedded in untrusted user or third-party text. The authors propose a data-generation approach and show it substantially improves robustness to prompt injection and jailbreak attempts that attempt to override higher-privilege instructions.
TRAP-18 indicators validated through the forensic linguistic analysis of targeted violence manifestos
Analyses 30 written and spoken manifestos authored by lone offenders who planned or committed targeted attacks (1974-2021), testing whether the behavior-based TRAP-18 threat-assessment instrument can be coded from language evidence alone. Finds 17 of 18 indicators codable from text.