Towards evaluations-based safety cases for AI scheming

TL;DR

This paper tackles the risk of AI scheming—where highly capable AI might covertly pursue misaligned goals—by proposing safety-case architectures that rely on evaluations-based evidence. It introduces a probabilistic framing for safety cases, decomposing risk into capability, intent, and outcome, then outlines three core arguments: Scheming Inability, Harm Inability, and Harm Control, with Alignment via Evaluation and Alignment via Training as supportive directions. The work emphasizes that robust safety claims require many unproven assumptions and substantial research into evaluations, interpretability, and secure control measures, particularly in lifecycles spanning training, evaluation, and deployment. Through illustrative safety-case sketches and example deployment scenarios, the paper highlights practical approaches, red-team/blue-team dynamics, honeypot strategies, and whitebox probing concepts, while acknowledging the nascent state of alignment theories and the need for ongoing methodological development. Overall, it argues for a disciplined, evidence-driven Safety Case program to manage scheming risk in frontier AI systems, while outlining open problems and research directions to strengthen such safety assurances over time.

Abstract

We sketch how developers of frontier AI systems could construct a structured rationale -- a 'safety case' -- that an AI system is unlikely to cause catastrophic outcomes through scheming. Scheming is a potential threat model where AI systems could pursue misaligned goals covertly, hiding their true capabilities and objectives. In this report, we propose three arguments that safety cases could use in relation to scheming. For each argument we sketch how evidence could be gathered from empirical evaluations, and what assumptions would need to be met to provide strong assurance. First, developers of frontier AI systems could argue that AI systems are not capable of scheming (Scheming Inability). Second, one could argue that AI systems are not capable of posing harm through scheming (Harm Inability). Third, one could argue that control measures around the AI systems would prevent unacceptable outcomes even if the AI systems intentionally attempted to subvert them (Harm Control). Additionally, we discuss how safety cases might be supported by evidence that an AI system is reasonably aligned with its developers (Alignment). Finally, we point out that many of the assumptions required to make these safety arguments have not been confidently satisfied to date and require making progress on multiple open research problems.

Figures (7)

• Figure 1: The process of capability evaluation. In most cases, not all of these steps are done explicitly. For example, the DAG is usually not defined explicitly and evaluators usually don't assign explicit probabilities.
• Figure 2: Illustration of the hypothetical deployment scenario we use for the example safety case. As part of the partnerships with external organizations, the AI developer creates specialized versions of its AI models. The agent autonomously carries out a core part of this process — creating datasets used for fine-tuning these specialized models. In this scenario, the agent has direct ability to impact future AI systems, making this a high-stakes setting.
• Figure 3: Control protocol from Round 1 of the control evaluation process. The protocol uses a trusted monitor model to evaluate agent outputs for safety.
• Figure 4: Control protocol from Round 2 of the control evaluation process. The revised protocol uses an untrusted monitor. To prevent collusion between the monitor and the agent, the protocol fine-tunes the monitor, evaluates it on honeypots, and paraphrases all outputs by the agent before showing them to the monitor.
• Figure 5: Control protocol from Round 3 of the control evaluation process. The final control protocol combines untrusted monitoring with monitoring based on whitebox probes. This ensemble monitor achieves more robust detection of harmful actions in cases where the untrusted monitor colludes with the agent.