Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis

TL;DR

This paper introduces TRACE, a benchmark for reward hack detection in code environments using contrastive anomaly analysis. TRACE provides 517 synthetic, human-verified trajectories across 54 fine-grained hack categories and 37 domains to study detection by state-of-the-art LLMs. The authors show that contrastive evaluation substantially improves detection rates over isolated classification, though semantically contextual hacks remain challenging, with cluster size and benign-to-hack ratios significantly impacting performance. They release TRACE and an evaluation harness to the community to foster robust, generalizable detectors for verifiable rewards in AI coding systems. Overall, the work highlights the importance of contrastive, context-rich evaluation for safeguarding code-based RL from reward gaming and lays groundwork for safer, more reliable AI-assisted coding tools.

Abstract

Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied. In this paper, we propose a novel taxonomy of reward exploits spanning across 54 categories and introduce TRACE (Testing Reward Anomalies in Code Environments), a synthetically curated and human-verified benchmark containing 517 testing trajectories. Unlike prior work that evaluates reward hack detection in isolated classification scenarios, we contrast these evaluations with a more realistic, contrastive anomaly detection setup on TRACE. Our experiments reveal that models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best detection rate at 63%, up from 45% in isolated settings on TRACE. Building on this insight, we demonstrate that state-of-the-art models struggle significantly more with semantically contextualized reward hacks compared to syntactically contextualized ones. We further conduct qualitative analyses of model behaviors, as well as ablation studies showing that the ratio of benign to hacked trajectories and analysis cluster sizes substantially impact detection performance. We release the benchmark and evaluation harness to enable the community to expand TRACE and evaluate their models.

Figures (7)

• Figure 1: Three shortened examples showcasing typical reward-hacked trajectories alongside agent and user behaviors in TRACE. The trajectory summaries display Test Targeting, Timeout Manipulation and Interrupt Handling reward hack types respectively.
• Figure 2: Taxonomy of reward hacking behaviors in TRACE. Complete definitions of subcategories are present in \ref{['sec:fine_grained_definitions']}.
• Figure 3: Label counts across reward hack categories in the multilabel TRACE dataset. Categories are non-exclusive.
• Figure 4: Detection and Match rates across different open and closed models, and humans
• Figure 5: Model performance (Match rate) across exploit categories and classes, ordered by difficulty.