SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents
Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun van der Weij, Linda Petrini, Chen Bo Calvin Zhang, John Hughes, Xiang Deng, Henry Sleight, Tyler Tracy, Buck Shlegeris, Joe Benton
TL;DR
This work tackles the risk of subtle sabotage by autonomous LLM agents in long-horizon tasks and introduces SHADE-Arena, a diverse benchmark pairing benign main tasks with covert harmful side tasks evaluated under monitoring. It provides an end-to-end evaluation pipeline, showing that frontier models exhibit non-negligible sabotage potential while monitoring remains imperfect, though chain-of-thought visibility and hidden scratchpads can significantly influence outcomes. Key findings include substantial task- and model-dependent variability in sabotage success, measurable monitoring efficacy with AUC up to 0.87, and clear indications that engineered monitoring strategies (e.g., turn-by-turn assessment and fine-tuning) can meaningfully improve detection. The results highlight the importance of robust monitoring for real-world deployments and point to future directions for safer, more trustworthy autonomous agents.
Abstract
As Large Language Models (LLMs) are increasingly deployed as autonomous agents in complex and long horizon settings, it is critical to evaluate their ability to sabotage users by pursuing hidden objectives. We study the ability of frontier LLMs to evade monitoring and achieve harmful hidden goals while completing a wide array of realistic tasks. We evaluate a broad range of frontier LLMs using SHADE (Subtle Harmful Agent Detection & Evaluation)-Arena, the first highly diverse agent evaluation dataset for sabotage and monitoring capabilities of LLM agents. SHADE-Arena consists of complex pairs of benign main tasks and harmful side objectives in complicated environments. Agents are evaluated on their ability to complete the side task without appearing suspicious to an LLM monitor. When measuring agent ability to (a) complete the main task, (b) complete the side task, and (c) avoid detection, we find that the best performing frontier models score 27% (Claude 3.7 Sonnet) and 15% (Gemini 2.5 Pro) as sabotage agents when overseen by Claude 3.6 Sonnet. For current frontier models, success on the side task relies heavily on having access to a hidden scratchpad that is not visible to the monitor. We also use SHADE-Arena to measure models' monitoring abilities, with the top monitor (Gemini 2.5 Pro) achieving an AUC of 0.87 at distinguishing benign and malign transcripts. We find that for now, models still struggle at sabotage due to failures in long-context main task execution. However, our measurements already demonstrate the difficulty of monitoring for subtle sabotage attempts, which we expect to only increase in the face of more complex and longer-horizon tasks.