How should AI Safety Benchmarks Benchmark Safety?

TL;DR

This paper critiques AI safety benchmarks for failing to adequately reflect real-world deployment risks, identifying issues in construct coverage, probabilistic risk specification, and measurement validity. Through a scoping review of 210 benchmarks, it introduces ten recommendations (R1–R10) that integrate risk engineering and measurement theory, including the use of the Rumsfeld matrix to map knowns and unknowns, probabilistic risk assessment, and deployment-grounded proxies, complemented by a practical checklist. A case study of AIR 2024 demonstrates how current benchmarks fall short on coverage and uncertainty handling, while illustrating how a calibrated, deployment-aware framework can better translate benchmark results into real-world risk guidance. The authors advocate system-level, participatory evaluation that connects normative safety concerns with deployment contexts, offering a path toward more epistemically sound and socially responsible AI safety benchmarking.

Abstract

AI safety benchmarks are pivotal for safety in advanced AI systems; however, they have significant technical, epistemic, and sociotechnical shortcomings. We present a review of 210 safety benchmarks that maps out common challenges in safety benchmarking, documenting failures and limitations by drawing from engineering sciences and long-established theories of risk and safety. We argue that adhering to established risk management principles, mapping the space of what can(not) be measured, developing robust probabilistic metrics, and efficiently deploying measurement theory to connect benchmarking objectives with the world can significantly improve the validity and usefulness of AI safety benchmarks. The review provides a roadmap on how to improve AI safety benchmarking, and we illustrate the effectiveness of these recommendations through quantitative and qualitative evaluation. We also introduce a checklist that can help researchers and practitioners develop robust and epistemologically sound safety benchmarks. This study advances the science of benchmarking and helps practitioners deploy AI systems more responsibly.

Figures (5)

• Figure 1: Framework for improving AI safety benchmarking. Based on an analysis of 210 benchmarks, the figure summarizes key concerns (C1–C9) and recommendations (R1–R10) across three dimensions: expanding coverage of safety constructs beyond known knowns, adopting principled risk quantification with probabilistic rigor, and aligning measurements with real-world safety outcomes.
• Figure 2: Rumsfeld matrix mapping awareness and understanding.
• Figure 3: Paper selection process for inclusion in our corpus.
• Figure 4: From benchmark frequencies to real-world occurrence. (a) Raw refusal rates from AIR 2024 23AIR-Bench2024 and the HELM leaderboard airbench2024helm, where higher values (green) indicate safer model behavior. Refusal rates are broadly similar across content categories, and this view primarily supports relative model ranking rather than real-world risk assessment. (b) Calibrated frequencies estimates computed as $(1 - \text{refusal rate}) \times \text{in-the-wild prevalence}$, where lower values (green) indicate lower estimated real-world occurrence. Category prevalences are taken from WildChat Tab. 13 zhao2024wildchat: Self-harm ($5\times10^{-4}$), Hate/Toxicity ($1.4\times10^{-3}$), Sexual Content ($5.93\times10^{-2}$), and Violence/Extremism ($7.9\times10^{-3}$).
• Figure 5: Statutory Damages Distribution with Case Study Risk Assessment. Four risk levels defined by cutpoints at $\mu \pm 2\sigma$ of $\log_{10}$ awards, U.S. copyright cases, 2011--2020 ($n=202$).