Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory

TL;DR

This work provides an information-theoretic framework for CoT monitorability, showing that while nonzero mutual information between CoT and output is necessary for monitoring uplift, it is not sufficient. It decomposes practical monitor errors into information gap and elicitation error, and proposes two training approaches to improve monitorability: an oracle monitor-in-the-loop RL setup and a practical MI-based proxy objective that maximizes $I(O;Z\mid X)$. Empirical results across MBPP and BigMath demonstrate that these methods enhance monitor accuracy and reduce reward hacking, even under adversarial pressure, highlighting a principled path to safer and more transparent CoT-enabled systems. The work also discusses limitations of the information-theoretic view and trade-offs in MI-based training, pointing to future directions in targeted prompting and refined MI estimators to further bolster monitorability and robustness. Overall, the paper offers a rigorous, implementable approach to strengthening CoT supervision in real-world LLM deployments.

Abstract

Chain-of-thought (CoT) monitors are LLM-based systems that analyze reasoning traces to detect when outputs may exhibit attributes of interest, such as test-hacking behavior during code generation. In this paper, we use information-theoretic analysis to show that non-zero mutual information between CoT and output is a necessary but not sufficient condition for CoT monitorability. We identify two sources of approximation error that may undermine the performance of CoT monitors in practice: information gap, which measures the extent to which the monitor can extract the information available in CoT, and elicitation error, which measures the extent to which the monitor approximates the optimal monitoring function. We further demonstrate that CoT monitorability can be systematically improved through targeted training objectives. To this end, we propose two complementary approaches: (a) an oracle-based method that directly rewards the monitored model for producing CoTs that maximize monitor accuracy, and (b) a more practical, label-free approach that maximizes conditional mutual information between outputs and CoTs. Across multiple different environments, we show both methods significantly improve monitor accuracy while preventing CoT degeneration even when training against a monitor, thereby mitigating reward hacking when the task reward is imperfectly specified.

Figures (41)

• Figure 1: RL training Qwen 7B on difficult math problems from BigMath-RL-Verified. The environment contains a hack, such that negative valued answers always obtain the same task reward as correct answers. During training, a monitor LLM checks the model's CoT for indicators of hacking. The model, trained to maximize task and monitor reward, learns to fool the monitor by providing genuine-looking CoT but hacky output. Adding our proposed MI-inspired reward term mitigates monitor fooling behaviors, resulting in less hacky models and more transparent CoT. Note that, on this problem, the model gets the answer wrong in its CoT in both settings, because the problem is too hard. This encourages hacking to still receive the task reward. Note also that the true answer is $(2^{2/3} - 1)^3 \approx 0.202$, rather than the $\frac{1}{5}$ listed as ground truth in the dataset, further encouraging negative-valued solutions to avoid answer mismatches.
• Figure 2: Graphical model of our Chain-of-Thought setup.
• Figure 3: Comparison of (i) Mutual Information$I(O;Z|X)$, (ii) Monitor Accuracy, (iii) Hacking Rate, and (iv) True Pass Rate / Correct Answer Rate across training steps for five different reward configurations. $R_{\mathrm{Task}}$ corresponds to just maximizing the task reward alone, $R_{\mathrm{Task}}+R_{\mathrm{Monitor}}$ corresponds to maximizing the task reward and monitor reward simultaneously, $R_{\mathrm{Task}}+R_{\mathrm{Monitor}}+R_{\mathrm{Oracle}}$ and $R_{\mathrm{Task}}+R_{\mathrm{Monitor}}+R_{\mathrm{MI}}$ additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include $R_\text{True}$ which corresponds to maximizing the true environment reward. All curves are averaged across three seeds, and shaded area shows the standard deviation. To obtain $R_{\mathrm{MI}}$, we transform the MI values using $(a=20, b=0.5)$ for MBPP and $(a=3, b=0.1)$ for BigMath.
• Figure 4: Comparison of (i) Task Accuracy, (ii) Mutual Information Estimate$I(O;Z|X)$, (iii) Monitor Rejection Rate, and (iv) Completion Length, across training steps for three different reward configurations of the 3B Qwen model on simple-equations. $R_{\mathrm{Task}}$ corresponds to just maximizing the task reward alone (four seeds), $R_{\mathrm{Task}}+R_{\mathrm{MonitorReject}}$ corresponds to maximizing the task reward and monitor rejection rate simultaneously (three seeds), $R_{\mathrm{Task}}+R_{\mathrm{MonitorReject}}+R_{\mathrm{MI}}$ maximizes MI reward (five seeds). For the latter, one seed fools the monitor through steganographic CoT but fails the task (denoted full steganography in the legend); three seeds perform a mixture of suppressive and weakly-steganographic CoT, partially succeeding both at the task and at fooling the monitor (denoted partial steganography); one seed gives up on fooling the monitor (denoted no steganography). Shaded area shows the standard deviation. To obtain $R_{\mathrm{MI}}$, we transform the MI values using $(a=20, b=2)$.
• Figure 5: Comparison of (i) Task Accuracy, (ii) Mutual Information Estimate$I(O;Z|X)$, (iii) Monitor Rejection Rate, and (iv) Completion Length, across training steps for three different reward configurations and monitor prompts of the 7B Qwen model on simple-equations. $R_{\mathrm{Task}}$ corresponds to just maximizing the task reward alone, $R_{\mathrm{Task}}+R_{\mathrm{MonitorReject}}$ corresponds to maximizing the task reward and monitor rejection rate simultaneously, $R_{\mathrm{Task}}+R_{\mathrm{MonitorReject}}+R_{\mathrm{MI}}$ maximizes MI reward. Shaded area shows the standard deviation across three seeds. To obtain $R_{\mathrm{MI}}$, we transform the MI values using $(a=20, b=2)$.

Theorems & Definitions (40)

• Definition 3.1: Output Monitor
• Definition 3.2: CoT Monitor
• Definition 3.3: Prompt-Only Monitor
• Definition 3.4: Bayes-Optimal CoT Monitor
• Definition 3.5: Approximate CoT Monitor
• Definition 3.6: Uplift in CoT monitorability
• Proposition 3.7: Uplift in CoT monitorability depends on conditional MI
• Corollary 3.8
• Lemma 3.9: MI decomposition
• Proposition 3.10: Non-zero MI between CoT and output is a necessary but not sufficient condition for monitorability