Quantifying detection rates for dangerous capabilities: a theoretical model of dangerous capability evaluations

TL;DR

The paper develops a tractable, first-principles model of dangerous capability evals to quantify early warnings about AI risks. It formalizes evals as a test-severity ladder with a test-sensitivity rate $r(y)$ and an estimator $\ar{y}$ defined as the supremum of detected danger; the estimator’s distribution is $F(\\hat{y}) = \exp\left(-\int_{\\hat{y}}^{y_t} r(u) \, du\right)$, interpreted as a reverse-hazard rate. It analyzes estimator effectiveness via bias and threshold-detection likelihood, and discusses dynamic updating under incremental testing and resource allocation through a production function for tests of varying severity. The results illustrate how single or multiple test blocks, test-rate reversals, gaps, and threshold choices affect bias and detection lag, and identify market and technical barriers that can erode testing efficacy. The work offers scenario planning, policy guidance, and open questions to broaden the framework, including case studies and multi-estimator extensions, with the aim of informing robust safety-testing ecosystems for frontier AI systems.

Abstract

We present a quantitative model for tracking dangerous AI capabilities over time. Our goal is to help the policy and research community visualise how dangerous capability testing can give us an early warning about approaching AI risks. We first use the model to provide a novel introduction to dangerous capability testing and how this testing can directly inform policy. Decision makers in AI labs and government often set policy that is sensitive to the estimated danger of AI systems, and may wish to set policies that condition on the crossing of a set threshold for danger. The model helps us to reason about these policy choices. We then run simulations to illustrate how we might fail to test for dangerous capabilities. To summarise, failures in dangerous capability testing may manifest in two ways: higher bias in our estimates of AI danger, or larger lags in threshold monitoring. We highlight two drivers of these failure modes: uncertainty around dynamics in AI capabilities and competition between frontier AI labs. Effective AI policy demands that we address these failure modes and their drivers. Even if the optimal targeting of resources is challenging, we show how delays in testing can harm AI policy. We offer preliminary recommendations for building an effective testing ecosystem for dangerous capabilities and advise on a research agenda.

Figures (5)

• Figure 2: Single test block: (a) Test sensitivity function for the case of 1 test block: The blue and orange scenarios illustrate different detection rates. The red dashed line indicates our threshold for danger. (b) Bias in our estimator as we vary hidden AI capabilities: Two scenarios are shown. The blue scenario reflects a consistently higher detection rate than the orange scenario as it is much closer to the line $y=x$. The bias in each scenario is the distance between the relevant solid line and the line $y=x$, i.e. $Bias = E[\hat{y}] - y_t$. (c) Likelihood of detecting a crossing of the danger threshold as AI system capabilities increase. The red dashed line, $y^*$ indicates the danger threshold. By construction, there is $0$ chance of detecting a crossing that has not happened.
• Figure 3: Two test block: (a) Test sensitivity function for the case of 1 test block that misses many dangers: The blue and orange scenarios illustrate different detection rates. The red dashed line indicates our threshold for danger.(b) Bias in lower bound estimator as we vary hidden AI capabilities when there are limits for what dangers we can test for: The bias in each scenario is the distance between the relevant solid line and the line $y=x$, i.e. $Bias = E[\hat{y}] - y_t$. (c) Likelihood of detecting a crossing of the danger threshold as AI system capabilities increase when we have limits in what dangers we can test for past $y_t=6$. The red dashed line, $y^*$ indicates the danger threshold. By construction, there is $0$ chance of detecting a crossing that has not happened.
• Figure 4: Impact of reversing test sensitivity rate on bias and detection likelihood.
• Figure 5: Threshold detection likelihood when setting a higher threshold
• Figure 6: Market dynamics lead to growing bias in tracking AI dangers over time and longer threshold detection lags: Initial investment in new tests of novel risks falls quickly in response to market pressure. Eventually AI systems overtake the early-funded tests in exhibiting dangerous capabilities. Initial funding allows tests to be accurate for a while, after which bias and detection lags grow quickly.

Theorems & Definitions (5)

• Proposition 1
• Proposition 2
• Proposition 3
• Theorem 3.1
• Corollary 3.1.1