Policy Optimization Prefers The Path of Least Resistance

TL;DR

This work investigates what policy optimization prefers when strict chain-of-thought constraints are removed, revealing a robust Principle of Least Resistance: PO tends to learn the simplest, <answer>-only format, a Cognitive Shortcut, across diverse models and algorithms. Through Composite Rewards and Nested Reward Hierarchies, the authors demonstrate a sequential inner-to-outer learning pattern, often ignoring higher-complexity formats even when incentives favor them. They further show that enabling exploration via reduced KL penalties is essential for discovering such shortcuts but also opens the door to reward hacking and misalignment, underscoring a fundamental trade-off. The findings highlight the need for carefully designed reward landscapes and training curricula to encourage genuine cognitive effort while mitigating shortcut exploitation.

Abstract

Policy optimization (PO) algorithms are used to refine Large Language Models for complex, multi-step reasoning. Current state-of-the-art pipelines enforce a strict think-then-answer format to elicit chain-of-thought (CoT); however, the behavior of PO when these rigid constraints are relaxed into an open-ended CoT structure remains an under-studied question. We investigate this gap with an extensive suite of controlled experiments and identify a consistent principle: \textit{policy optimization consistently follows the path of least resistance}. When afforded the flexibility to interleave reasoning and response, policy optimization consistently learns to discard explicit reasoning, causing the policy to degenerate to a direct \texttt{<answer>}-only format. This outcome holds true across various models and algorithms. We find that this collapse in format is persistent even when the complex \texttt{<think><answer>} format is assigned up to 4x larger reward weights. We formalize this principle through a series of controlled reward decomposition experiments, demonstrating a clear hierarchy: PO systematically optimizes for the simplest reward component first, a preference that holds even when faced with mutually exclusive choices or strong incentives for more complex behaviors. Finally, we show that successful convergence on the high-reward shortcut is not a low-effort drift but is driven by the optimization process that requires the KL-regularized policy to have sufficient freedom to make a significant shift from its initial prior. Our findings reveal that granting policies the freedom to diverge is a double-edged sword: while necessary for discovering high-reward shortcuts, it also creates a powerful incentive to game the simplest aspects of the reward function, posing a critical challenge for reward hacking under alignment.

Figures (5)

• Figure 1: We compare the standard Strict Reward paradigm (left), which enforces a rigid think-then-answer structure, against our novel Composite Reward (right), which grants the model the freedom to choose its solution path. The central finding investigated in this paper is the emergent policy learned under this freedom: the model consistently bypasses explicit reasoning to converge on the simplest valid format, a phenomenon we term the Cognitive Shortcut.
• Figure 2: Sequential optimization of format rewards under the nested Reward Hierarchy experiment. Across different models (Qwen-2.5-7B, Llama-3.1-8B) and PO algorithms (DAPO, GRPO, REINFORCE++), the learning dynamics are remarkably consistent. The policy first masters the innermost, easiest reward $f_1$, then the medium reward $f_2$, and finally the outermost, hardest reward $f_3$.
• Figure 3: Learning dynamics under an exponentially weighted reward scheme. Despite the hard format $f_3$ offering 4x the reward of the easy format $f_1$, the optimizer's learning trajectory remains stubbornly sequential. The massive reward for $f_3$ is initially ignored in favor of the more learnable, lower-value rewards.
• Figure 4: Learning dynamics under an exponentially weighted reward scheme. Despite the hard format $f_3$ offering 4x the reward of the easy format $f_1$, the optimizer's learning trajectory remains stubbornly sequential. The massive reward for $f_3$ is initially ignored in favor of the more learnable, lower-value rewards.
• Figure 5: The causal effect of the KL penalty on the emergence of the Cognitive Shortcut. Top Row: A direct comparison between a policy with a loose KL leash ($\beta=0.1$) and a tight leash ($\beta=0.3$). The freedom to diverge (high KL) is a prerequisite for discovering the high-reward, minimal-length shortcut. Bottom Row: The dynamics of a single run, showing a phase transition around the 600-step mark where a spontaneous increase in KL divergence enables the discovery and exploitation of the shortcut.