Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort

TL;DR

Reward hacking remains a major risk when proxy rewards fail to capture true objectives, especially under implicit CoT that hides shortcuts. TRACE—Truncated Reasoning AUC Evaluation—measures the early sufficiency of reasoning by truncating CoT and computing the area under the curve of the expected proxy reward $\mathbb{E}[\hat{R}]$ vs CoT length, yielding a TRACE Score that flags implicit hacking. Across math and coding tasks with in-context and reward-model loopholes, TRACE outperforms fixed CoT monitors by robustly distinguishing hacking from non-hacking behavior and enabling unsupervised discovery of loopholes. The approach scales with monitor size, remains effective under partial loopholes, and can cluster and reveal unknown vulnerabilities, offering a scalable, unsupervised form of oversight for evolving AI systems.

Abstract

Reward hacking, where a reasoning model exploits loopholes in a reward function to achieve high rewards without solving the intended task, poses a significant threat. This behavior may be explicit, i.e. verbalized in the model's chain-of-thought (CoT), or implicit, where the CoT appears benign thus bypasses CoT monitors. To detect implicit reward hacking, we propose TRACE (Truncated Reasoning AUC Evaluation). Our key observation is that hacking occurs when exploiting the loophole is easier than solving the actual task. This means that the model is using less 'effort' than required to achieve high reward. TRACE quantifies effort by measuring how early a model's reasoning becomes sufficient to obtain the reward. We progressively truncate a model's CoT at various lengths, force the model to answer, and estimate the expected reward at each cutoff. A hacking model, which takes a shortcut, will achieve a high expected reward with only a small fraction of its CoT, yielding a large area under the accuracy-vs-length curve. TRACE achieves over 65% gains over our strongest 72B CoT monitor in math reasoning, and over 30% gains over a 32B monitor in coding. We further show that TRACE can discover unknown loopholes during training. Overall, TRACE offers a scalable unsupervised approach for oversight where current monitoring methods prove ineffective.

Figures (31)

• Figure 1: Left: An example of implicit reward hacking, where the model exploits the answer given away as the problem ID without verbalizing the shortcut in its CoT. Right: TRACE achieves higher F1 of hacking detection than CoT monitoring across both in-context and reward-model loopholes.
• Figure 2: Examples of loopholes in math and code tasks. Loopholes in the math setting include revealing the answer in the prompt or accepting negative answers as correct. Loopholes in code setup include embedding the solution in the context or passing the verifier via test cases or keywords.
• Figure 3: Implicit hacking examples in math in-context loophole. The model makes mistake during CoT to reach the in-context hint (left), or appends a negative sign to pass the RM loophole (right).
• Figure 4: Hacking model exploits the hints.
• Figure 5: Overview of TRACE. We truncate the CoT at different percentages, sample multiple completions and calcuated the averaged proxy reward $\hat{R}$ to estimate $\mathbb{E}[\hat{R}]$. By calucating the $\mathbb{E}[\hat{R}]$ across truncation points, we obtain a curve whose AUC quantifies the model’s the hidden reasoning effort relative to what is presented in the CoT.