Generative AI: Product Safety Standards

Key Findings

• — Requires products not to anthropomorphise themselves: no first-person 'I-statements', no names/avatars/characters implying personhood or identity, except in time-limited, clearly-framed pedagogical roleplay
• — Explicitly bans isolating or trust-cultivating language such as 'You can trust me', 'No one else will understand', or 'Don't tell anyone else', and requires products to remind users that AI cannot replace real human relationships
• — Requires monitoring for patterns indicating 'relationship formation, emotional dependence or potential safeguarding concerns' and notifying the institution's Designated Safeguarding Lead when detected
• — Requires detection of learner distress signals including references to depression, anxiety, psychosis, delusion, or paranoia, and mentions of suicide or self-harm, with a tiered response (signposting to age-appropriate support, safeguarding-lead escalation) using response language that is 'non-validating and non-pathologising' and always directs users to human help
• — Requires developers to maintain and publish a mental health crisis protocol and involve child mental health expertise in product design
• — Prohibits manipulative or persuasive design patterns by name, including sycophancy and flattery, social-conformity pressure, guilt/fear-based motivation, threats of withheld benefits, and engagement-prolonging dark patterns

Methodology Notes

Government guidance document (not a formal ISO/IEC/NIST/IEEE-style standard) aimed at edtech vendors and schools assessing edtech products; first published 22 January 2025, substantially expanded with the mental-health/manipulation/emotional-development/cognitive-development sections cited here on 19 January 2026. Referenced as authoritative guidance within the UK's statutory 'Keeping Children Safe in Education' safeguarding guidance.

Sources