Peer-reviewed Authoritative

WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs

Presents WildGuard, an open moderation tool for large language models that jointly detects harmful intent in prompts, safety risks in responses, and model refusal. It is released with WildGuardMix, a training and evaluation dataset of about 92,000 labeled examples across 13 risk categories that includes adversarial jailbreaks and matched refusal/compliance pairs. Published at NeurIPS 2024 (Datasets and Benchmarks track).

Read at arxiv.org Archived copy

Publisher

Allen Institute for AI (AI2)

Published

26 Jun 2024

Added

today

DOI

—

Key Findings

Handles three moderation tasks in one model: prompt-harm detection, response-harm detection, and refusal detection
WildGuardMix pairs plain and adversarial-jailbreak prompts with labeled responses across 13 risk categories (~92,000 examples)
As a moderator it substantially reduces reported jailbreak success rates and improves refusal detection over prior open tools by up to 26.4%

Methodology Notes

Peer-reviewed (NeurIPS 2024 Datasets and Benchmarks track); primary version on arXiv. Benchmark comparisons against other open tools are the authors' own.

Sources

arXiv preprint (NeurIPS 2024 D&B) (primary)

Archived snapshot (Wayback Machine) — preserved against link rot

Topics

Guardrails & moderation Benchmarks Red-teaming Eval methodology

Authors

Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, Nouha Dziri

Cite This

APA

Seungju Han et al. (2024). WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. Allen Institute for AI (AI2). https://arxiv.org/abs/2406.18495