Reinforcement Learning Towards Broadly and Persistently Beneficial Models

Key Findings

• — Beneficial-trait RL improved performance on more than 80% of over 50 independent out-of-distribution alignment and beneficial-behaviour benchmarks compared with a compute-matched baseline
• — Evaluations that improved include reward hacking, deception, harmful advice, specification compliance, health, mental health and safety suites not used in training
• — Training restricted to a single domain (health) still produced broad improvements on non-health evaluations
• — Models trained this way were harder to steer toward harmful behaviour with adversarial prompts or subsequent fine-tuning
• — The work is framed as the beneficial counterpart of emergent misalignment, where narrow training on problematic behaviour generalizes broadly

Methodology Notes

Vendor research on OpenAI models: a dataset of realistic multi-domain situations scored for beneficial traits, mixed into a broader post-training distribution and trained with RL; evaluation on more than 50 public and internal benchmarks; adversarial persistence tested with prompts and fine-tuning. Internal benchmarks and model identities are not fully disclosed. Blog post dated 18 June 2026; arXiv 2606.24014 v1 submitted 22 June 2026. Verified by fetching the blog post and the arXiv abstract page (both HTTP 200). A coverage miss from June, entered on the alignment-blog enumeration.

Sources