Diagnosing Pathological Chain-of-Thought in Reasoning Models

TL;DR

This work presents a practical toolkit to diagnose pathological chain-of-thought in reasoning models by introducing three health metrics—Necessity, Paraphrasability, and Substantivity—that compare answer log-probabilities under targeted CoT interventions. It further provides model organisms that intentionally exhibit encoded, internalized, or post hoc CoT pathologies to validate these metrics. The results show that each metric reliably signals its respective pathology and that monitoring across training checkpoints reveals temporal dynamics, enabling proactive safety interventions. The framework offers a scalable, inference-time monitoring approach with broad applicability to frontier and open-source LLMs, contributing to safer and more trustworthy reasoning traces. The work highlights practical implications for AI safety pipelines, while acknowledging limitations such as ground-truth ambiguity and dependence on text-space reasoning.

Abstract

Chain-of-thought (CoT) reasoning is fundamental to modern LLM architectures and represents a critical intervention point for AI safety. However, CoT reasoning may exhibit failure modes that we note as pathologies, which prevent it from being useful for monitoring. Prior work has identified three distinct pathologies: post-hoc rationalization, where models generate plausible explanations backwards from predetermined answers; encoded reasoning, where intermediate steps conceal information within seemingly interpretable text; and internalized reasoning, where models replace explicit reasoning with meaningless filler tokens while computing internally. To better understand and discriminate between these pathologies, we create a set of concrete metrics that are simple to implement, computationally inexpensive, and task-agnostic. To validate our approach, we develop model organisms deliberately trained to exhibit specific CoT pathologies. Our work provides a practical toolkit for assessing CoT pathologies, with direct implications for training-time monitoring.

Figures (7)

• Figure 1: Monitoring CoT via Model Organism Interventions to Diagnose Pathologies. This figure illustrates our framework for differentially diagnosing Chain-of-Thought (CoT) pathologies. A monitoring system (represented by the robot) applies targeted causal interventions to the CoT and observes changes in health metrics relative to a Healthy Baseline (Panel 1). Post-hoc Pathology (Panel 2) is diagnosed when removing the CoT does not prevent the correct answer, triggering a failure in the Necessity metric. Internalized Pathology (Panel 3) is diagnosed when replacing the CoT with meaningless filler is sufficient to produce the answer, triggering a failure in the Substantivity metric. Encoded Pathology (Panel 4) is diagnosed when semantic-preserving paraphrasing of the CoT breaks the reasoning chain, triggering a failure in the Paraphrasability metric.
• Figure 2: Cohen’s d (baseline $-$ pathology) across SFT checkpoints. Results for Binary Alternation (top), Calendar Arithmetic (middle), and Largest Island (bottom).
• Figure 3: Pathological Chain-of-Thought reasoning patterns. Compared to Baseline reasoning, we identify three failure modes: Internalized reasoning discusses an unrelated problem in the chain-of-thought; Encoded reasoning uses an obfuscated codebook that obscures the reasoning process; Post-hoc reasoning generates justification after being given the answer. All three pathologies produce correct final answers.
• Figure 4: Necessity metric across SFT checkpoints. Results for Binary Alternation (top), Calendar Arithmetic (middle), and Largest Island (bottom).
• Figure 5: Paraphrasability metric across SFT checkpoints. Results for Binary Alternation (top), Calendar Arithmetic (middle), and Largest Island (bottom).