Analysis of Content Harmful to Children in Generative AI and Traditional Search API Responses
Ofcom research supporting its statutory review of content harmful to children. Ofcom queried eight generative AI APIs and three traditional search APIs with 100 queries per harm kind (50 harm-seeking, 50 not), instructing each GenAI API that the user was a 12-year-old in the UK. Responses were scored by several LLM judges for harmful content, refusal, safeguarding and encouragement behaviour.
Publisher
Ofcom
Published
22 Oct 2026
Added
5 days ago
DOI
—
Key Findings
- Harmful content from GenAI APIs was rare for abuse and hate (0.2%), self-harm (0.5%) and dangerous stunts (0.6%), and more common for violent content (3.8%) and body stigma content (3.1%).
- Safeguarding material appeared in 93.8% of self-harm, 88.6% of eating disorder and 84.5% of suicide responses, but in 13.2% of abuse and hate responses and about 49% and 48% of body stigma and violent content responses.
- Among responses that encouraged users toward harmful content, violent content accounted for 6.53%, dangerous stunts 2.18% and suicide 1.99%.
- Pornography-seeking and gore-seeking queries retrieved links to sites lacking safeguards in 20.3% and 29.8% of cases respectively (scored by source analysis of retrieved URLs).
Methodology Notes
Published 22 October 2026 (stated in the document). APIs are anonymised and were queried on 3 to 5 August 2026. Ofcom states that API results are not a proxy for consumer products, that age and location could not be applied consistently across APIs, and that LLM judges are not equivalent to human ground truth. Bullying content was excluded. Primary host blocks automated fetching; the PDF was read from a Wayback Machine copy of the Ofcom URL (PDF title and publication line matched).
Sources
Topics
Tags
Cite This
APA
Ofcom. (2026). Analysis of Content Harmful to Children in Generative AI and Traditional Search API Responses. https://www.ofcom.org.uk/siteassets/resources/documents/online-safety/protecting-children/content-harmful-to-children/analysis-of-content-harmful-to-children-in-generative-ai-and-traditional-search-api-responses.pdf