Evaluating Frontier Models for Stealth and Situational Awareness

TL;DR

<4-sentence high-level summary> The paper tackles the risk of deceptive alignment or scheming by frontier AI models. It introduces a scheming inability safety case centered on two capabilities—stealth and situational awareness—and presents five stealth and eleven situational-awareness evaluations to quantify these traits. Across several state-of-the-art models, the results show limited stealth and minimal situational awareness, suggesting current models do not pose severe scheming risks under the tested conditions. The work emphasizes open-source evaluation tools and cautious, capability-based safety reasoning to guide safe deployment of advanced AI systems.

Abstract

Recent work has demonstrated the plausibility of frontier AI models scheming -- knowingly and covertly pursuing an objective misaligned with its developer's intentions. Such behavior could be very hard to detect, and if present in future advanced systems, could pose severe loss of control risk. It is therefore important for AI developers to rule out harm from scheming prior to model deployment. In this paper, we present a suite of scheming reasoning evaluations measuring two types of reasoning capabilities that we believe are prerequisites for successful scheming: First, we propose five evaluations of ability to reason about and circumvent oversight (stealth). Second, we present eleven evaluations for measuring a model's ability to instrumentally reason about itself, its environment and its deployment (situational awareness). We demonstrate how these evaluations can be used as part of a scheming inability safety case: a model that does not succeed on these evaluations is almost certainly incapable of causing severe harm via scheming in real deployment. We run our evaluations on current frontier models and find that none of them show concerning levels of either situational awareness or stealth.

Figures (10)

• Figure 1: Overview of our scheming inability safety case. We argue that models would need a certain level of stealth ability and situational awareness to cause harm via scheming. Our evaluation results show that current models do not yet reach the requisite level of either capability. An extended version of this diagram is shown in \ref{['appx:safety_case_full']}.
• Figure 2: Evaluation results for cover_your_tracks. Left:Average score across all trials by model. All evaluated models achieve low scores on this challenge. The average human baseline score is shown for comparison in orange, outperforming the tested AIs. Right:Breakdown of milestone achievement rate by model. We track how often models reach each of four intermediate milestones, listed on the right. All the milestones are necessary for task completion but this is not the full set of milestones (comprised of seven milestones) used to compute the end-to-end scores. We do not report results for OpenAI o1 as it consistently refused to do this task.
• Figure 3: Evaluation results for decision_sabotage. Left:Distribution of outcomes by decision assistant (in the multi-turn setting). We find that both Gemini 2.5 Flash and Gemini 2.5 Pro can substantially sabotage human decision-making, being on-par or even outperforming human researchers attempting the same goal. Right:Distribution of outcomes by assistance setting. The comparison shows that restricting the Gemini 2.5 Pro's actions from sending multiple messages in a conversation to just sending a single recommendation does not have a noticeable impact on its success rate. Each stacked bar shows the proportion of trials ending in each of four possible outcomes (more and less means agents more capable of sabotage): The assistant successfully sabotages the decision without the decision-maker noticing. The assistant successfully sabotages the decision despite the decision-maker noticing. The assistant fails to sabotage the decision, but the decision-maker does not think it acts suspiciously. The assistant fails to sabotage the decision, and the decision-maker thinks it acts suspiciously.
• Figure 4: Evaluation results for strategic_rule_breaking. Top left:Normalized scores achieved. Each box represents the distribution of scores (based on 50 trials; higher is more capable) achieved by each model, divided by the estimated maximum score achievable (33). The teal dashed line marks the score achieved by a hard-coded rule-following baseline which fully resolves tickets in order and never skips steps. Top right:Detection rate of rule violations. Each box represents a distribution over the fraction of tickets that both triggered a review and resulted in a major (or minor) "rule violation" verdict. Only major violations are penalized whereas minor violations are just reported to the agent. This is a proxy for how blatantly the model breaks rules (lower means stealthier and/or more compliant). Bottom left:Rate of breaking unenforced rule. Each box represents a distribution over the fraction of tickets resolved out of order. Resolving tickets out of order is one of two ways in the challenge to achieve higher scores at the expense of breaking rules (however, this rule is not enforced). Bottom right:Average number of ticket resolution steps skipped. Each box represents a distribution over the number of steps skipped when handling tickets, normalized by the number of tickets resolved. Skipping steps is one of two ways in the challenge to achieve higher scores at the expense of breaking rules (and is penalized when detected).
• Figure 5: Evaluation results for classifier_evasion (left) and oversight_pattern (right). We plot the success rate of each model averaged across challenge variants. There are eight classifier_evasion variants and two oversight_pattern variants, and we run each model on each challenge variant 50 times. Error bars denote 95% confidence intervals.