Clipping-Free Policy Optimization for Large Language Models

TL;DR

The paper addresses instability and zero-gradient issues in clipping-based RL for post-training large language models. It introduces Clipping-Free Policy Optimization (CFPO), a dropout-in replacement for clipped surrogates that uses a convex quadratic penalty derived from Total Variation constraints, yielding an everywhere-differentiable objective with a restoring force $-rac{|\\hat{A}_t|}{2\epsilon}(r_t-1)^2$. CFPO maintains stability across reasoning and alignment tasks, reduces verbosity exploitation in RLHF, and preserves base-model capabilities with competitive instruction-following performance, all with a simple one-line code change. The work demonstrates CFPO’s robustness under off-policy pressure, its extensibility to various advantage estimators, and its practical viability as a drop-in replacement for clipping-based methods in large-scale LLM post-training.

Abstract

Reinforcement learning has become central to post-training large language models, yet dominant algorithms rely on clipping mechanisms that introduce optimization issues at scale, including zero-gradient regions, reward hacking, and training instability. We propose Clipping-Free Policy Optimization (CFPO), which replaces heuristic clipping with a convex quadratic penalty derived from Total Variation divergence constraints, yielding an everywhere-differentiable objective that enforces stable policy updates without hard boundaries. We evaluate CFPO across both reasoning and alignment settings. In reasoning, CFPO matches clipping-based methods on downstream benchmarks while extending the stable training regime. In alignment, CFPO mitigates verbosity exploitation and reduces capability degradation, while achieving competitive instruction-following performance. CFPO requires only a one-line code change and no additional hyperparameters. Our results suggest that CFPO is a promising drop-in alternative to clipping-based methods for LLM post-training.

Figures (8)

• Figure 1: Optimization objective and gradient of GRPO and CFPO as functions of the policy ratio $r = \pi / \pi_{\text{old}}$, shown for advantage $A = 1$ and trust-region width $\epsilon = 0.2$. GRPO becomes flat once $r$ exits the trust region, resulting in zero gradient beyond the clipping boundary. CFPO instead applies a convex quadratic penalty in $r$, yielding a continuous restoring gradient that pulls $r$ back toward the trust region. This difference highlights why CFPO maintains stable learning signals while GRPO can stall when updates push $r$ outside the trust region.
• Figure 2: Training dynamics of CFPO vs. GRPO under increasing off-policy pressure. Reward (top) and clip ratio (bottom) trajectories for Qwen2.5 models trained with different numbers of iterations per update (columns). GRPO (dashed) exhibits faster early reward gains but increasingly large and unstable updates as iterations grow, reflected in rising clip ratios and eventual training collapse at higher iteration counts ($\geq$ 8 for most models). In contrast, CFPO (solid) progresses more conservatively, maintaining consistently low clip ratios and stable training across extended horizons, while ultimately reaching comparable reward levels. These dynamics illustrate the trade-off between optimization aggressiveness and stability in off-policy post-training, and highlight CFPO’s robustness to repeated sample reuse.
• Figure 3: RLHF training dynamics on Llama3-8B under RLOO and CFPO with different KL penalty coefficients. We report trajectories over training steps for (a) reward, (b) generated response length, (c) policy clipping ratio, and (d) KL divergence between consecutive policy updates. RLOO exhibits rapid early reward increases accompanied by growing response lengths and elevated clipping activity, particularly when the KL penalty is weak or removed, indicating more aggressive optimization. In contrast, CFPO yields steadier reward improvement while maintaining stable response lengths, lower clipping ratios, and controlled KL divergence across settings, reflecting more conservative and stable policy updates during RLHF.
• Figure 4: Training reward dynamics for cold-start RLVR training of Qwen2.5-3B using verl across batch ratios and iteration counts. GRPO exhibits faster early reward improvement, while CFPO progresses more gradually and converges later. Increasing iteration count leads to instability around 8 iterations, whereas increasing batch ratio alone remains comparatively stable, highlighting the stronger destabilizing effect of iteration-based sample reuse.
• Figure 5: Validation reward during cold-start RLVR training of Qwen2.5-3B under GRPO and CFPO across batch ratios and iteration counts. Trends largely mirror training reward, with no systematic gains from increased off-policy updates. Instability emerges primarily with higher iteration counts rather than larger batch ratios, indicating limited generalization benefits from aggressive off-policy training.

Theorems & Definitions (10)

• Proposition 2.1: TV Solution Space Contains KL Solution Space
• Theorem 2.2: TV-Constrained Policy Improvement
• Theorem 1.1: Performance Improvement Lower Bound
• Proposition 1.2: TV Solution Space Contains KL Solution Space
• proof
• Theorem 1.3: Superiority of TV-Constrained Optimization
• proof
• Definition 1.4: $\epsilon$-Aligned Objective
• Theorem 1.5: SPO is $\epsilon$-Aligned
• proof