Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents
Preprint that builds a child-centred harm taxonomy and a coded corpus (KidAID) of 1,331 publicly reported AI incidents in which a minor plausibly experienced harm, drawn from the OECD.AI incident monitor, AIAAIC and the AI Incident Database. Each incident carries one individual-level harm code, one population-level harm code and one primary sociotechnical failure mode. The authors report that coded harm concentrates in representational/dignity and psychological categories and co-occurs most often with deployment-context, governance and misuse failures rather than model defects, and argue that model-level benchmarks and per-output filtering cannot reach the pathways most often documented for children.
Publisher
The University of Texas at Austin (Responsible AI Initiative), with Allen Institute for AI and Stanford University (arXiv preprint)
Published
26 Oct 2026
Added
today
Key Findings
- Corpus of 1,331 incidents: 1,143 from the OECD.AI incident monitor, 141 from AIAAIC and 47 from the AI Incident Database; counts were low through the mid-2010s, rose from about 2020 and accelerated after 2022
- Most common individual-level harm categories: Representation, Dignity and Social Identity (n=475, of which sexualization n=316 and sexual exploitation n=131) and Psychological harms (n=409, led by addiction/compulsive use n=79, self-harm n=69 and health deterioration n=66); 112 incidents (8.4%) could not be assigned an individual-level harm
- Human, organizational and contextual deployment failures were the primary failure mode in 779 incidents, led by violation of contextual norms (n=398), governance or regulatory noncompliance (n=227) and overreliance or misleading anthropomorphic design (n=146, 49 of them with a Psychological harm)
- Misuse/abuse failures accounted for 297 incidents (malicious use n=228); model behaviour or technical failures for 196 (general robustness n=128, hallucination/fabrication n=47)
- Inter-rater agreement on a 150-incident sample was Krippendorff's alpha 0.73 for harm categories and 0.67 for failure modes (kappa 0.58 for worst-case flags); the authors treat failure-mode findings as provisional
Methodology Notes
Post-hoc coding of publicly reported incidents; inclusion required material AI involvement, at least one affected minor (under 18) and enough detail to code harm and failure. Codebook seeded from adult-general taxonomies (Weidinger et al., Shelby et al.) with child-specific harm types; two annotators per incident with consensus resolution. Descriptive counts and co-occurrences only, no models or hypothesis tests; one primary failure mode per incident, so contributing failures are not counted. The incident scope is broad (generative depiction, recommender amplification, school surveillance, toys and voice assistants as well as chatbots); prevalence is frequency within public reporting, which the authors say favours acute, legally actionable events over slow-accumulating harms such as dependency. Privacy-minimized annotations and codebook at github.com/The-Responsible-AI-Initiative/KidAID (repository created 2026-10-18). Not peer reviewed. Verified from arXiv abs and HTML v1 and the GitHub API.
Sources
arXiv abstract page(opens in a new tab) (primary)
arXiv HTML v1(opens in a new tab) (26 Oct 2026)
KidAID annotations and codebook(opens in a new tab)
Archived snapshot (Wayback Machine)(opens in a new tab) — preserved against link rot
Topics
Authors
Junfeng Jiao, Kevin Chen, Shrestha Mishra, Abhejay Murali, David Atkinson, Shriti Raj, Saleh Afroogh
Tags
Cite This
APA
Junfeng Jiao et al. (2026). Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents. The University of Texas at Austin (Responsible AI Initiative), with Allen Institute for AI and Stanford University (arXiv preprint). https://arxiv.org/abs/2610.34705