ETR: Outcome-Guided Elastic Trust Regions for Policy Optimization

TL;DR

The paper identifies a fundamental limitation of GRPO—static trust-region bounds that fail to account for heterogeneous, outcome-driven learning signals in RLVR. It introduces Elastic Trust Regions (ETR), a signal-aware mechanism that adjusts micro-level and macro-level clipping based on advantage magnitudes and group variances, effectively creating an implicit curriculum. The method yields higher accuracy and more stable exploration on math reasoning benchmarks, including AMC23 and AIME, and generalizes better to out-of-distribution tasks, while incurring negligible computational overhead. The results demonstrate that aligning optimization constraints with signal quality can substantially improve learning efficiency and policy robustness in reasoning-focused LLM post-training.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an important paradigm for unlocking reasoning capabilities in large language models, exemplified by the success of OpenAI o1 and DeepSeek-R1. Currently, Group Relative Policy Optimization (GRPO) stands as the dominant algorithm in this domain due to its stable training and critic-free efficiency. However, we argue that GRPO suffers from a structural limitation: it imposes a uniform, static trust region constraint across all samples. This design implicitly assumes signal homogeneity, a premise misaligned with the heterogeneous nature of outcome-driven learning, where advantage magnitudes and variances fluctuate significantly. Consequently, static constraints fail to fully exploit high-quality signals while insufficiently suppressing noise, often precipitating rapid entropy collapse. To address this, we propose \textbf{E}lastic \textbf{T}rust \textbf{R}egions (\textbf{ETR}), a dynamic mechanism that aligns optimization constraints with signal quality. ETR constructs a signal-aware landscape through dual-level elasticity: at the micro level, it scales clipping boundaries based on advantage magnitude to accelerate learning from high-confidence paths; at the macro level, it leverages group variance to implicitly allocate larger update budgets to tasks in the optimal learning zone. Extensive experiments on AIME and MATH benchmarks demonstrate that ETR consistently outperforms GRPO, achieving superior accuracy while effectively mitigating policy entropy degradation to ensure sustained exploration.

Figures (8)

• Figure 1: Not all learning signals deserve equal trust. ETR replaces GRPO’s fixed threshold with dynamic bounds that expand at the macro-level for valuable samples and contract at the micro-level for uncertain steps.
• Figure 2: Comparison of optimization landscapes between GRPO and ETR. Left (GRPO): the static clipping range suffers from static mismatch, where high-value signals (green stars, large advantage) are aggressively truncated, limiting efficient learning. Right (ETR): our method creates a signal-aware adaptive boundary. By scaling the trust region proportional to signal strength, ETR successfully encapsulates both high-advantage samples while constraining negative signals (red triangles) to ensure stability.
• Figure 3: A case study on a high-difficulty reasoning step. We visualize the policy ratio trajectory for a critical step that requires a large update ($\approx 1.38$). (a) GRPO (limit 1.20) clips the gradient greatly, losing information. (b) & (c) Applying only Micro or Macro adjustments alleviates this issue, but still results in some clipping. (d) Full ETR combines both adjustments, raising the dynamic boundary to 1.40 to capture the peak learning signal. This demonstrates how ETR prevents the loss of crucial information in important sparse-reward reasoning tasks.
• Figure 4: Val Accuracy on AIME 2024/2025 using Qwen3-8B-Base. Standard GRPO (yellow) suffers from performance collapse in later learning stages. ETR (blue) maintains a steady upwards trend in both Mean@32 and Best@32.
• Figure 5: Policy Entropy Evolution. GRPO collapses to zero, limiting exploration. Clip-High leads to high, unstable entropy. ETR maintains healthy entropy levels, facilitating sustained learning.