Rethinking the Trust Region in LLM Reinforcement Learning

TL;DR

The paper argues that PPO's ratio clipping is ill-suited for LLM fine-tuning due to long-tailed vocabularies, and introduces Divergence Proximal Policy Optimization (DPPO), a trust-region method that directly constrains distributional divergence (e.g., $D_{TV}$ or $D_{KL}$) using memory-efficient Binary and Top-K approximations. It provides a finite-horizon policy-improvement bound tailored to LLM generation, showing how a principled divergence-based constraint improves stability and learning efficiency. Extensive scaling experiments across multiple models and tasks demonstrate DPPO's superior stability and faster convergence compared to PPO-based baselines, with reduced training-inference mismatch. The work offers practical guidelines and theoretical grounding for RL-based LLM fine-tuning, highlighting the importance of appropriately constraining updates to avoid over-penalizing rare tokens while guarding against destabilizing mass shifts in common tokens.

Abstract

Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that the core ratio clipping mechanism in PPO is structurally ill-suited for the large vocabularies inherent to LLMs. PPO constrains policy updates based on the probability ratio of sampled tokens, which serves as a noisy single-sample Monte Carlo estimate of the true policy divergence. This creates a sub-optimal learning dynamic: updates to low-probability tokens are aggressively over-penalized, while potentially catastrophic shifts in high-probability tokens are under-constrained, leading to training inefficiency and instability. To address this, we propose Divergence Proximal Policy Optimization (DPPO), which substitutes heuristic clipping with a more principled constraint based on a direct estimate of policy divergence (e.g., Total Variation or KL). To avoid huge memory footprint, we introduce the efficient Binary and Top-K approximations to capture the essential divergence with negligible overhead. Extensive empirical evaluations demonstrate that DPPO achieves superior training stability and efficiency compared to existing methods, offering a more robust foundation for RL-based LLM fine-tuning.

Figures (17)

• Figure 1: Comparison of PPO and the proposed DPPO (the Binary-TV variant in \ref{['sec:method_binary']}). (Left) The surrogate objective and corresponding masks for PPO and DPPO. PPO (and variants like GRPO) employs a heuristic mask based on the probability ratio, which over-penalizes low-probability tokens and under-penalizes high-probability ones (\ref{['sec:method_limitations']}). In contrast, DPPO utilizes a more principled mask based on a direct approximation of policy divergence (e.g., Total Variation), ensuring updates stay within a theoretically grounded trust region (\ref{['sec:llm_tr']}). (Right) Experimental results on the AIME24 using Qwen3-30B-A3B-Base. DPPO significantly outperforms GRPO baselines, achieving superior training efficiency and stability even without rollout routing replay (R3) (\ref{['sec:scaling_exp']}).
• Figure 2: The plots show numerical differences between a training and an inference engine for Qwen3-30B-A3B-Base with identical parameters. (Left) The probability ratio (used in PPO) is highly volatile for low-probability tokens. (Right) In contrast, the TV divergence (used in DPPO) is more stable. This highlights a key flaw of PPO's clipping mechanism: it over-penalizes low-probability tokens, which can slow down learning; and under-penalizes high-probability tokens, which can permit large, destabilizing updates.
• Figure 3: DPPO variants achieve stable training while controlling the training-inference mismatch at a low level. In contrast, methods without a trust region (PG-IS, CISPO) or with a misspecified one (MiniRL) suffer from growing mismatch and eventual collapse.
• Figure 4: Switching the stable DPPO-KL to a decoupled objective causes the mismatch to grow and performance to collapse, confirming that the trust region must be anchored to the rollout policy.
• Figure 5: Isolating the source of instability. The solid curves are training rewards, while the dashed lines are the percentage of bad updates. Starting with the unstable PG-IS, applying a minimal mask that only blocks large-divergence bad updates on negative samples is sufficient to stabilize training, indicating these bad updates are the primary cause of training instability.

Theorems & Definitions (7)

• Theorem 2.1
• Theorem 3.1: Performance Difference Identity for LLMs
• Theorem 3.2: Policy Improvement Bound for LLMs
• proof : Proof of \ref{['lem:llm_identity']}
• Lemma 2.1: Bound on Sequence-Level TV Divergence
• proof
• proof : Proof of \ref{['thm:llm_tr_bound']}