TRE: Encouraging Exploration in the Trust Region

TL;DR

The paper tackles the failure of global entropy regularization for training LLMs in RL by identifying cumulative tail risk in long-horizon generation. It introduces Trust Region Entropy (TRE), which confines exploration to a model-defined trust region and comes in two variants, TRE-K and TRE-P, enabling local entropy maximization within a plausible token subset. Through extensive experiments on MATH, Countdown, and HH alignment tasks with Qwen bases ranging from 1.5B to 7B, TRE consistently outperforms vanilla PPO, standard Entropy, and KL-Cov baselines, with TRE-P often delivering the strongest gains. TRE reduces overconfidence and tail-induced noise, improving reasoning stability and alignment performance, thereby offering a robust approach to exploration in LLM RL settings. The proposed method has practical implications for safer, more reliable LLM fine-tuning and alignment, particularly in reasoning-intensive and preference-alignment scenarios.

Abstract

Entropy regularization is a standard technique in reinforcement learning (RL) to enhance exploration, yet it yields negligible effects or even degrades performance in Large Language Models (LLMs). We attribute this failure to the cumulative tail risk inherent to LLMs with massive vocabularies and long generation horizons. In such environments, standard global entropy maximization indiscriminately dilutes probability mass into the vast tail of invalid tokens rather than focusing on plausible candidates, thereby disrupting coherent reasoning. To address this, we propose Trust Region Entropy (TRE), a method that encourages exploration strictly within the model's trust region. Extensive experiments across mathematical reasoning (MATH), combinatorial search (Countdown), and preference alignment (HH) tasks demonstrate that TRE consistently outperforms vanilla PPO, standard entropy regularization, and other exploration baselines. Our code is available at https://github.com/WhyChaos/TRE-Encouraging-Exploration-in-the-Trust-Region.

Figures (4)

• Figure 1: Impact of Maximum Generation Length ($T$) on Entropy Regularization Efficacy. Experiments are conducted using Qwen2.5-1.5B-Instruct. The left $y$-axis displays the performance difference ($\Delta$) between entropy-regularized PPO and the vanilla baseline. The grey dashed line (corresponding to the right $y$-axis) depicts the absolute accuracy of the vanilla baseline. Across both domains, mild entropy regularization helps in short-horizon settings but becomes detrimental as the horizon extends, showing its incompatibility with long-context reasoning.
• Figure 2: Overview of Trust Region Entropy (TRE). Standard entropy regularization (red line) indiscriminately flattens the distribution, causing significant probability mass to leak into the vast tail of invalid tokens (Tail Risk). In contrast, TRE (green line) increases diversity exclusively within the Trust Region $\mathcal{A}_{\text{TR}}$ while maintaining the suppression of the Tail.
• Figure 3: Evolution of Average Peak Probability ($\bar{P}_{\max}$) on Qwen2.5-1.5B-Instruct. The curves illustrate the policy's confidence level during training across three tasks. The Vanilla PPO baseline (Red) consistently trends toward saturation (high confidence), while TRE-K (Blue) and TRE-P (Gold) maintain a lower peak probability. A smoothing factor of 0.8 is applied for visualization clarity.
• Figure 4: Hyperparameter Analysis on Qwen2.5-1.5B-Instruct. We compare the performance of size-based trust regions ($K$), nucleus-based trust regions ($P$), and the Min-Ent baseline ($K=1$ boundary). The dashed red line represents the Vanilla (PPO) performance without entropy regularization.