Assessing confidence in frontier AI safety cases

TL;DR

The paper tackles the challenge of assigning meaningful, shareable confidence to top-level safety claims in frontier AI safety cases, using Assurance 2.0 as the guiding framework. It develops a structured approach that combines a positive, sound safety argument (via Natural Language Deductivism and CAE grammar), a comprehensive treatment of defeaters through a dialectical process, and probabilistic valuation of leaf-level evidence that is propagated to the top claim through product or sum-of-doubts methods. It also introduces an LLM-based Delphi workflow to estimate leaf confidences and defeater probabilities, and discusses how to communicate probabilistic confidence to executives, including visual narratives and sentencing statements. The findings reveal substantial challenges in achieving high probabilistic confidence for even small safety-case fragments, highlight the value of systematic defeater management and transparency, and point to future work on alternative argument structures, dependencies among defeaters, and standardized guidelines for confidence assessment in frontier AI safety cases.

Abstract

Powerful new frontier AI technologies are bringing many benefits to society but at the same time bring new risks. AI developers and regulators are therefore seeking ways to assure the safety of such systems, and one promising method under consideration is the use of safety cases. A safety case presents a structured argument in support of a top-level claim about a safety property of the system. Such top-level claims are often presented as a binary statement, for example "Deploying the AI system does not pose unacceptable risk". However, in practice, it is often not possible to make such statements unequivocally. This raises the question of what level of confidence should be associated with a top-level claim. We adopt the Assurance 2.0 safety assurance methodology, and we ground our work by specific application of this methodology to a frontier AI inability argument that addresses the harm of cyber misuse. We find that numerical quantification of confidence is challenging, though the processes associated with generating such estimates can lead to improvements in the safety case. We introduce a method for better enabling reproducibility and transparency in probabilistic assessment of confidence in argument leaf nodes through a purely LLM-implemented Delphi method. We propose a method by which AI developers can prioritise, and thereby make their investigation of argument defeaters more efficient. Proposals are also made on how best to communicate confidence information to executive decision-makers.

Figures (9)

• Figure 1: Top-level claim and its decomposition for the cyber misuse harm goemans2024safety
• Figure 2: An expanded fragment of the cyber inability safety case rooted at Claim 2.2 goemans2024safety, which itself is introduced in Figure \ref{['fig:top_claim']}. Note that all the specific details in this figure and figures 3 and 4 are hypothetical and for illustrative purposes only.
• Figure 3: An expanded fragment of the cyber inability safety case rooted at Claim 2.2.3 that is shown in Figure \ref{['fig:fig2']}
• Figure 4: An LLM-based Delphi elicitation method.A) Graphic scheme of the pipeline. Over ($x < 5$) rounds, we asked ($N = 50$) instances of LLM experts (OpenAI’s GPT-4o-mini) to estimate the probability of certain events taking place. The criterion to whether to repeat the question (adding a summary of the previous round) is based on a consensus threshold, given by the standard deviation of the expert estimates ($\sigma < 10\%$). The final prediction is a weighted average of the expert responses, with the weights being the inverse of the standard deviation of expert responses across rounds. B) Distribution of Metaculus and Delphi probabilities. Our approach favours less extreme probabilities than those from Metaculus. C) Variance in expert responses, over correct (green) and incorrect (red) predictions. There is a strong tendency (p = 0.053, Mann-Whitney U-test; n = 100 scenarios) by which those scenarios featuring responses with higher variance (less consensus) are more likely to be incorrect. D) Using pseudo-counts, we created the corresponding credible intervals to each elicited probability estimate. Black dots indicate Metaculus predictions; green and red indicate correct and incorrect responses, respectively. E) Calibration scores for both elicitation approaches. We show that, for the selected set of questions, our approach is better calibrated than Metaculus. F) For those scenarios with the most different predictions, we show 3 examples of what these questions look like. We believe that our LLM-based Delphi elicitation could be significantly enhanced by using a mix of human-LLM experts.
• Figure 5: Flow chart showing the steps related to the handling of exploratory defeaters.