The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Introduces an instruction-hierarchy training method that teaches LLMs to prioritize system/developer-level instructions over conflicting instructions embedded in untrusted user or third-party text. The authors propose a data-generation approach and show it substantially improves robustness to prompt injection and jailbreak attempts that attempt to override higher-privilege instructions.
Publisher
OpenAI
Published
19 Apr 2024
Added
today
DOI
—
Key Findings
- LLMs by default often treat system-prompt instructions with the same priority as text from untrusted users or third parties, creating an override vulnerability
- A synthetic data-generation method that trains models to selectively honor higher-privilege instructions substantially reduces successful prompt-injection and jailbreak attacks in evaluation
- The method generalizes to instruction types not seen during training
Methodology Notes
Lab technical report (arXiv preprint, no separate peer-reviewed venue identified as of this record). Introduces the instruction-hierarchy framework and reports evaluation results against held-out and out-of-distribution attack types.
Sources
arXiv preprint (primary)
Archived snapshot (Wayback Machine) — preserved against link rot
Authors
Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, Alex Beutel
Tags
Cite This
APA
Eric Wallace et al. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. OpenAI. https://arxiv.org/abs/2404.13208