What AI evaluations for preventing catastrophic risks can and cannot do

TL;DR

The paper critically examines AI capability evaluations, arguing they provide concrete lower bounds on what systems can do and can illuminate misuse risks under substantial effort, while also contributing to scientific understanding and governance discussions. However, fundamental limits persist: evaluations cannot establish upper bounds, reliably forecast future capabilities, or robustly assess misalignment and autonomy risks, and they cannot foresee unknown unknown risks. The authors advocate a cautious, multi-faceted approach to safety that uses evaluations as one tool among others, emphasizing incremental governance measures such as third-party audits, conservative red lines, defense-in-depth cybersecurity, continuous monitoring, and sustained research. The work underscores the need to avoid over-reliance on evaluations as guarantees of safety and to develop complementary strategies that address the deep, structural uncertainties inherent in frontier AI systems.

Abstract

AI evaluations are an important component of the AI governance toolkit, underlying current approaches to safety cases for preventing catastrophic risks. Our paper examines what these evaluations can and cannot tell us. Evaluations can establish lower bounds on AI capabilities and assess certain misuse risks given sufficient effort from evaluators. Unfortunately, evaluations face fundamental limitations that cannot be overcome within the current paradigm. These include an inability to establish upper bounds on capabilities, reliably forecast future model capabilities, or robustly assess risks from autonomous AI systems. This means that while evaluations are valuable tools, we should not rely on them as our main way of ensuring AI systems are safe. We conclude with recommendations for incremental improvements to frontier AI safety, while acknowledging these fundamental limitations remain unsolved.

Figures (2)

• Figure 1: Evaluations are performed such that the regions assumed to be safe are always overlapping. In the ideal case this maintains continuous safety coverage. Blue circles indicate evaluations, while black bars show the scaling range where models are assumed safe based on evaluation results. The fading of the black bar represents decreasing confidence in safety as scaling continues.
• Figure 2: Precursor-based capability forecasting and potential failure modes. Blue circles indicate evaluations, while black bars show the scaling range where models are assumed safe based on evaluation results. (A) Intended scenario: Precursor capabilities are detected early enough to implement safety measures before dangerous capabilities emerge. (B) Evaluations too infrequent failure: The gap between precursor and dangerous capabilities is smaller than expected, leading to dangerous capabilities emerging before the next scheduled evaluation. (C) No warning period failure: Dangerous capabilities emerge without developing the expected precursor capabilities, leaving no warning period for implementing safeguards.