Output Supervision Can Obfuscate the Chain of Thought

TL;DR

The paper identifies feedback spillover as a fundamental risk when training against output monitors, showing that such supervision can obfuscate chain-of-thought reasoning instead of preserving it. It provides a formal, gradient-based analysis that separates conditional and parametric spillover mechanisms and introduces two mitigations: reward targeting to cut CoT-gradient flow and a Mind–Face two-model setup to decouple reasoning from presentation. Empirically, spillover is observed across polynomial derivative factoring, hints-based QA, and multi-turn reasoning tasks, with the mitigations achieving Pareto improvements in monitorability and task performance in several cases, though not universally. The work argues that frontier-model development should explicitly test for spillover and adopt robust mitigation strategies to maintain CoT monitorability in safety-critical deployments.

Abstract

OpenAI (2025) showed that training against a chain of thought (CoT) monitor can cause obfuscated CoTs, which contain bad behavior the monitor cannot detect. They proposed to keep CoTs monitorable by training only against output monitors that do not have access to CoT. We show that such training can still cause obfuscated CoTs via two mechanisms. First, when a model is trained to produce a safe-looking output, that model may generalize to making its CoTs look safe. Second, since later tokens are conditioned on earlier ones, safe-looking CoTs may increase the likelihood of safe outputs, causing safe-looking CoTs to be reinforced. We introduce two mitigations to address these two issues, which achieve a Pareto improvement in terms of monitorability and task performance compared to regular training.

Figures (7)

• Figure 1: Feedback spillover: training against an output monitor obfuscates the CoT.
• Figure 2: Our mitigations for feedback spillover.
• Figure 3: Results for the polynomial derivative factoring environment. Mind & Face with reward targeting significantly reduces feedback spillover while maintaining a high task reward.
• Figure 4: Results for the question answering environment with MMLU problems. Reward targeting partially mitigated feedback spillover without reducing the task reward. Other mitigations destabilized training.
• Figure 5: Results for the question answering environment with difficult math problems. Reward targeting and Mind & Face partially mitigated feedback spillover, but had a higher output penalty than when no mitigations were used. Applying both mitigations simultaneously destabilized training.

Theorems & Definitions (6)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6