Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods
Markov Grey, Charbel-Raphaël Segerie
TL;DR
Safety by Measurement presents a three‑part taxonomy for AI safety evaluation—what properties to measure (capabilities, propensities, and control), how to measure them (behavioral and internal techniques), and how results inform governance through structured frameworks and gates. It argues that benchmarks alone cannot ensure safety at frontier scales and details a comprehensive suite of evaluations, including dangerous capability, propensity, and control assessments, augmented by scaffolding, red teaming, supervised fine‑tuning, and mechanistic interpretability. The paper surveys formal frameworks (Model Organisms Framework; RSP, Preparedness, Frontier Safety), design considerations (affordances, automation, audits), and domain‑specific evaluations (cybersecurity, deception, autonomous replication, long‑term planning, situational awareness) to translate measurement into actionable safety policies. It also discusses critical limitations—proving absence, sandbagging, safetywashing, unknown unknowns, and governance gaps—and advocates for independent auditing and governance‑driven decision rules to manage scaling risk. Collectively, the work provides a central reference for researchers, policymakers, and practitioners seeking to ground AI safety in robust, scalable evaluation practices that align with responsible development and deployment.
Abstract
As frontier AI systems advance toward transformative capabilities, we need a parallel transformation in how we measure and evaluate these systems to ensure safety and inform governance. While benchmarks have been the primary method for estimating model capabilities, they often fail to establish true upper bounds or predict deployment behavior. This literature review consolidates the rapidly evolving field of AI safety evaluations, proposing a systematic taxonomy around three dimensions: what properties we measure, how we measure them, and how these measurements integrate into frameworks. We show how evaluations go beyond benchmarks by measuring what models can do when pushed to the limit (capabilities), the behavioral tendencies exhibited by default (propensities), and whether our safety measures remain effective even when faced with subversive adversarial AI (control). These properties are measured through behavioral techniques like scaffolding, red teaming and supervised fine-tuning, alongside internal techniques such as representation analysis and mechanistic interpretability. We provide deeper explanations of some safety-critical capabilities like cybersecurity exploitation, deception, autonomous replication, and situational awareness, alongside concerning propensities like power-seeking and scheming. The review explores how these evaluation methods integrate into governance frameworks to translate results into concrete development decisions. We also highlight challenges to safety evaluations - proving absence of capabilities, potential model sandbagging, and incentives for "safetywashing" - while identifying promising research directions. By synthesizing scattered resources, this literature review aims to provide a central reference point for understanding AI safety evaluations.