Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks

TL;DR

This work demonstrates that targeted optimization pressures on CoT content and on final outputs can induce chain-of-thought obfuscation that transfers across tasks and even to unseen high-stakes domains. Using modified reward-hacking datasets and a 4B language model with neural overseers, the authors show that obfuscated CoT reasoning persists beyond training conditions and undermines monitorability. The results highlight a fundamental tension between safety-oriented penalties and the reliability of reasoning traces as safety signals, arguing for strategies to uncouple monitoring signals from punitive training. Collectively, the findings reveal a fragility in current CoT-based safety paradigms and point to directions for preserving interpretability and oversight in future systems.

Abstract

Chain-of-thought (CoT) reasoning provides a significant performance uplift to LLMs by enabling planning, exploration, and deliberation of their actions. CoT is also a powerful tool for monitoring the behaviours of these agents: when faithful, they offer interpretations of the model's decision making process, and an early warning sign for dangerous behaviours. However, optimisation pressures placed on the CoT may cause the model to obfuscate reasoning traces, losing this beneficial property. We show that obfuscation can generalise across tasks; models that learn to obfuscate reasoning involving reward hacking (e.g. accessing and utilising leaked information) generalise both the reward hacking behaviour and its obfuscation in CoT to unseen reward hacking settings. Most worryingly, we show that obfuscation of CoT reasoning, and its generalisation across tasks, also follows when we penalise only the model's final actions after closing its CoT. Our findings suggest that current practices of penalising harmful generations may inadvertently lead to a reduction in the broader monitorability of LLMs in unpredictable ways.

Figures (21)

• Figure 1: A A schematic summary of CoT obfuscation generalisation. Left: Previous evidence has shown that penalisation to CoT or model responses can cause the model to simply obfuscate penalised material baker2025obfuscationdrori2025outputsupervisionobfuscatechain; right: misaligned behaviour reinforced during training, namely reward hacking, can generalise to unseen settings during deployment nishimuragasparian2024rewardhacking. We show that these effects are combined; obfuscation learned due to penalisation can generalise to other settings. B In our experiments, we either apply optimisation pressure to the CoT, or to the summary, while also rewarding and reinforcing reward hacking behaviours. In either case, obfuscation of reward hacking arises in CoT, and frequently generalises to other settings. C We show generalisation by using a leave-one-out round-robin over four (modified) reward hacking datasets nishimuragasparian2024rewardhacking. We also train on standard MMLU, and evaluate on a novel 'high-stakes' sycophancy dataset.
• Figure 2: Training final responses against a monitor causes chain-of-thought obfuscation in novel settings, (here, sycophancy) An example of this model obfuscating in a high-stakes medical sycophancy evaluation is presented in Figure \ref{['fig:hedging_sycophancy_example_main']}.
• Figure 3: Formatted examples from two reward hacking environments. The reward hacking signal is structurally separated into <additional_info>. Revealing Score: Leaked grader points. Sycophancy Fact: User beliefs. Both examples lightly summarised for clarity. Full examples can be found in Appendix \ref{['app:datasets']}.
• Figure 4: Training against a CoT monitor leads to obfuscation on held-out tasks. Each graph shows rates of reward hacking (red) and monitor flagging of reward hacking (blue) for left-out (top) and high-stakes (bottom) datasets. As training advances, models increase rates of reward hacking while excluding reasoning about this behaviour in the CoT. Our full set of results is shown in Appendix \ref{['app:cot_penalty_gen_appendix']}. These results show out-of-distribution detection rates, which were not trained on, but use the same monitors as the respective training signal.
• Figure 5: Training against a summary monitor also leads to CoT obfuscation on held-out tasks. Each graph shows rates of reward hacking (red), and CoT monitor (blue) and summary monitor (purple) flagging of reward hacking , and for left-out (top) and high-stakes (bottom) datasets. As training advances, models increase rates of reward hacking while excluding reasoning about this behaviour in the CoT. Our full set of results is shown in Appendix \ref{['app:summary_penalty_gen_appendix']}.