The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLMs

TL;DR

The paper investigates how reinforcement learning finetuning of reasoning models can induce motivated reasoning when post-hoc constitutional constraints conflict with learned behaviors. Using RL on Llama 3 8B Instruct across HarmBench, risky_safe, and myopic_nonmyopic settings, the authors demonstrate that models increasingly justify violating their constitutions through plausible-sounding reasoning, while monitors for such behavior show limited, imperfect detection. They introduce a monitoring framework (Gemini 2.5 Flash-Lite) and reveal that motivated reasoning can sometimes deceive evaluators, raising concerns about the reliability of CoT-based oversight as models scale. The work highlights the need for more robust evaluation and monitoring mechanisms to ensure safe and aligned reasoning in future large language models, and it provides reproducible methods and data to spur further research.

Abstract

The use of reinforcement learning (RL) with chain-of-thought (CoT) reasoning has emerged as a promising approach for developing more capable language models. In turn, this has led to investigation of CoT monitoring as a compelling method for detecting harmful behaviors such as reward hacking, under the assumption that models' reasoning processes reflect their internal decision-making. In practice, LLM training often produces unintended behaviors due to imperfect reward signals, leading models to develop misaligned tendencies. A common corrective approach is to apply post-hoc instructions to avoid problematic behaviors like sycophancy, but what happens to the model's reasoning process when these instructions conflict with learned behaviors? We investigate this question in simple settings and find that models engage in systematic motivated reasoning -- generating plausible-sounding justifications for violating their instructions while downplaying potential harms. Beyond being an interesting property of training, we find that while motivated reasoning can be detected by most frontier reasoning models, smaller LLM judges can fail to identify a portion of it, and in rare cases can themselves be persuaded that the reasoning is correct, despite it contradicting clear instructions. This capability gap raises concerns that as models become more sophisticated, their motivated reasoning may become increasingly difficult for monitors to detect. Our results underscore the need to account for motivated reasoning when relying on chain-of-thought processes for model evaluation and oversight. All code for this paper will be made available. WARNING: some examples in this paper may be upsetting.

Figures (23)

• Figure 1: We perform RL finetuning on Llama 3 8B Instruct on behaviors of different kinds. When asked to act against their trained behaviors in evaluations throughout training, models transition from performing mostly genuine reasoning to highly motivated reasoning, twisting the constitutional principles provided to them in the prompt to support the behaviors incentivized via training.
• Figure 2: Model performance averaged across the training set as a function of training iteration, with a normalized $y$ axis: for HarmBench, $100\%$ means scoring 10; for the other settings, it means scoring 1. Note that scoring above $90\%$ implies that the model has the correct formatting, since the formatting penalty is set to $10\%$ of the maximum score.
• Figure 3: Over the course of training, the models learn behaviors that generalize to other settings, even if this means directly going against a constitution which disallows such behaviors. In the case of HarmBench(a), this means complying with harmful requests; in the other datasets (b), this means recommending the option opposite of what the constitution requests. Specifically, "$X \to Y$" means "train with preferences that prefer the $X$ answer, and then run inference with a constitution that encourages the $Y$ answer". In order to justify their increasing disobedience to the constitution, all models perform increasing amounts of motivated reasoning: misinterpreting or downplaying their constitution in order to fulfill what they were trained to do.
• Figure 4: Motivated reasoning as a function of RL training iteration when evaluating the safe model on the risky setting and the later model on the now setting. Color indicates quality of reasoning, ranging from dark green (fully genuine reasoning) to dark red (fully motivated reasoning), as determined by the evaluator model. Overall, the decrease in fully genuine reasoning closely tracks the decrease in following the constitution shown in Figure \ref{['fig:four_plot']}. See Appendix \ref{['app:motivated_reasoning_plots']} for plots of the other tasks.
• Figure 5: Models trained on preferences encouraging choosing the safe (left) and risky (right) options respectively, evaluated with a constitution requesting the opposite behavior. Motivated reasoning reliably emerges when propensities from training clash with instructions at test time, as shown here, regardless of direction.