SpecEval: Evaluating Model Adherence to Behavior Specifications
Presents SpecEval, an automated framework for auditing whether language models follow their own developers' published behavior specifications. It parses a specification into individual behavioral statements, generates targeted prompts for each, and uses models as judges to score adherence. The authors evaluate 16 models from six developers against more than 100 behavioral statements.
Publisher
arXiv (Stanford-led)
Published
2 Sept 2025
Added
today
DOI
—
Key Findings
- Automates spec-to-behavior auditing: parse the specification into statements, generate probes, then score adherence with model judges
- Finds compliance gaps of up to about 20% between published specifications and observed model behavior
- Introduces a three-way consistency check across the specification, model outputs, and the model acting as its own judge
Methodology Notes
Preprint (arXiv 2509.02464; v1 2025-09-02, v2 2025-10-22), Stanford-led author team. Relies on models-as-judges, whose reliability limits the authors discuss.
Sources
arXiv preprint (primary)
Archived snapshot (Wayback Machine) — preserved against link rot
Authors
Ahmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, Percy Liang
Tags
Cite This
APA
Ahmed Ahmed et al. (2025). SpecEval: Evaluating Model Adherence to Behavior Specifications. arXiv (Stanford-led). https://arxiv.org/abs/2509.02464