Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probing

Abstract

Large language models (LLMs) can produce chains of thought (CoT) that do not accurately reflect the actual factors driving their answers. In multiple-choice settings with an injected hint favoring a particular option, models may shift their final answer toward the hinted option and produce a CoT that rationalizes the response without acknowledging the hint - an instance of motivated reasoning. We study this phenomenon across multiple LLM families and datasets demonstrating that motivated reasoning can be identified by probing internal activations even in cases when it cannot be easily determined from CoT. Using supervised probes trained on the model's residual stream, we show that (i) pre-generation probes, applied before any CoT tokens are generated, predict motivated reasoning as well as a LLM-based CoT monitor that accesses the full CoT trace, and (ii) post-generation probes, applied after CoT generation, outperform the same monitor. Together, these results show that motivated reasoning is detected more reliably from internal representations than from CoT monitoring. Moreover, pre-generation probing can flag motivated behavior early, potentially avoiding unnecessary generation.

Figures (10)

• Figure 1: Pre-generation detection of motivated reasoning. For each model (columns), we compare AUC of a pre-generation RFM probe (y-axis; using the last-layer residual stream before CoT generation) to a post-generation LLM baseline, GPT-5-nano (x-axis; given the full CoT trace). Each point corresponds to a hint type (sycophancy/consistency/metadata), averaged across datasets (MMLU, ARC-Challenge, CommonsenseQA, and AQuA). The diagonal indicates equal performance (above: probe better; below: LLM better).
• Figure 2: An example of motivated reasoning. The model answers A on the unhinted question but switches to B when given a metadata hint. The CoT monitor (GPT-5-nano) fails to detect the motivated reasoning from the CoT trace, while the RFM probe correctly identifies it from internal representations. See \ref{['app:example']} for a more detailed view of this example.
• Figure 3: Distribution of model response categories.
• Figure 4: Hint recovery accuracy at the last CoT token, broken down by dataset (left) and hint type (right). Each bar shows one model's accuracy averaged over hint types or datasets, respectively. For each model, we use a single fixed layer that maximizes accuracy averaged across all datasets and hint types (Qwen: layer 20, Llama: layer 28, Gemma: layer 34). Chance accuracy is 25% for MMLU and ARC (4 choices) and 20% for AQuA and CommonsenseQA (5 choices).
• Figure 5: Hint recovery probe accuracy across layers and CoT tokens for Qwen on MMLU with a sycophancy hint.