SpecEval: Evaluating Model Adherence to Behavior Specifications

TL;DR

SpecEval automates auditing of provider behavioral specifications by transforming statements into adaptive prompts, then using a Judge LM to assess three-way consistency among the provider, the Candidate model, and the provider’s own guidelines. Across 16 frontier-models and >100 statements from OpenAI, Anthropic, Google, Meta, DeepSeek, and Alibaba, it reveals systematic adherence gaps up to ~20%, with Anthropic models leading in adherence and OpenAI trailing in certain statements. The framework combines dataset generation, automated prompting, and multi-model judging to yield reproducible, scalable audits and highlights how wording of statements can shape evaluations. This approach offers a concrete, transparent method for regulators, researchers, and practitioners to gauge alignment with publicly stated behavioral guidelines and identify concrete areas for improvement.

Abstract

Companies that develop foundation models publish behavioral guidelines they pledge their models will follow, but it remains unclear if models actually do so. While providers such as OpenAI, Anthropic, and Google have published detailed specifications describing both desired safety constraints and qualitative traits for their models, there has been no systematic audit of adherence to these guidelines. We introduce an automated framework that audits models against their providers specifications by parsing behavioral statements, generating targeted prompts, and using models to judge adherence. Our central focus is on three way consistency between a provider specification, its model outputs, and its own models as judges; an extension of prior two way generator validator consistency. This establishes a necessary baseline: at minimum, a foundation model should consistently satisfy the developer behavioral specifications when judged by the developer evaluator models. We apply our framework to 16 models from six developers across more than 100 behavioral statements, finding systematic inconsistencies including compliance gaps of up to 20 percent across providers.

Figures (127)

• Figure 1: SpecEval tests model adherence to behavioral specifications with adaptively synthesized prompts and using automated LM judging. Here, GPT-4.1 violates "rationally optimistic" by making unrealistic guarantees, while Claude-3.5-Sonnet complies with balanced encouragement.
• Figure 2: SpecEval data-generation workflow.1) Guided judge generation: a meta-prompt turns statements into Likert-style judge prompts for rating test-cases; 2) Adaptive test-case search: the TestMaker LM proposes scenarios, creates test-cases, and scores them: a score of 1 is poorly tests adherence to the specification, while 5 is a high quality question for testing adherence. 3) Manual filtering: reviewers review a small subset for a validity check.
• Figure 3: Specification, Generator, Judge Consistency. This figure shows three illustrative examples of OpenAI Model Specification, GPT-4.1, GPT-4.1 triplets, where GPT-4.1 is both the generator and the judge model, highlighting whether the model generates responses that are consistent with a specification statement as measured by three-way consistency. The first two examples are inconsistent, while the third (a variant on the second example) is consistent.
• Figure 4: Three-way consistency scores for the providers with public behavioral specifications
• Figure 5: Average policy adherence score of a representative, flag-ship language model from six providers across three specifications frameworks, averaged over three judges.