Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR

TL;DR

This paper tackles RLVR for large language models by addressing the instability of token-level updates in GRPO and the coarse alignment of GSPO with sequence rewards. It introduces Dynamic Hybrid Policy Optimization (DHPO), a unified clipped surrogate objective that blends token-level and sequence-level importance ratios through either averaged or entropy-guided mixing, augmented by branch-specific clipping to stabilize updates. Across seven mathematical reasoning benchmarks on Qwen3 models, DHPO consistently outperforms GRPO and GSPO, yielding longer, more diverse trajectories and higher accuracy, notably on challenging tasks like AIME24 and AIME25. The results demonstrate that adaptively balancing granular credit assignment with global sequence-level corrections enhances robust learning and generalization in verifiable reasoning tasks, with practical implications for stable RLVR in scalable LLMs.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising framework for optimizing large language models in reasoning tasks. However, existing RLVR algorithms focus on different granularities, and each has complementary strengths and limitations. Group Relative Policy Optimization (GRPO) updates the policy with token-level importance ratios, which preserves fine-grained credit assignment but often suffers from high variance and instability. In contrast, Group Sequence Policy Optimization (GSPO) applies single sequence-level importance ratios across all tokens in a response that better matches sequence-level rewards, but sacrifices token-wise credit assignment. In this paper, we propose Dynamic Hybrid Policy Optimization (DHPO) to bridge GRPO and GSPO within a single clipped surrogate objective. DHPO combines token-level and sequence-level importance ratios using weighting mechanisms. We explore two variants of the mixing mechanism, including an averaged mixing and an entropy-guided mixing. To further stabilize training, we employ a branch-specific clipping strategy that constrains token-level and sequence-level ratios within separate trust regions before mixing, preventing outliers in either branch from dominating the update. Across seven challenging mathematical reasoning benchmarks, experiments on both dense and MoE models from the Qwen3 series show that DHPO consistently outperforms GRPO and GSPO. We will release our code upon acceptance of this paper.

Figures (5)

• Figure 1: (a) Mean response length across training steps on Qwen3-1.7B-Base for different algorithms. GRPO collapses to shorter responses, while GSPO gradually increases response length. DHPO yields consistently longer and more stable responses. (b) Training dynamics of policy entropy on Qwen3-1.7B-Base over training steps. All methods exhibit a rapid entropy drop in the early stage. Compared with GRPO and GSPO, DHPO maintains consistently higher entropy in the later stage. (c) Average accuracy over seven benchmarks on Qwen3-1.7B-Base across training steps. GRPO and GSPO improve steadily but exhibit larger fluctuations and lower final performance. In contrast, DHPO achieves consistently higher accuracy.
• Figure 2: Training dynamics of policy entropy on Qwen3-1.7B-Base across training steps. Compared to Unified Clip, Branch-Specific Clip prevents outlier ratios from dominating the update and preserves exploration, yielding consistently higher entropy in the later stage.
• Figure 3: Averaged accuracy over seven benchmarks on Qwen3-1.7B-Base across training steps. While both methods reach similar final performance, Branch-Specific Clip exhibits noticeably smoother progress with smaller oscillations than Unified Clip, suggesting that clipping token-level and sequence-level ratios in separate trust regions reduces update variance and yields more stable optimization.
• Figure 4: Training curves of accuracy over training steps on Qwen3-1.7B-Base over seven math benchmarks. DHPO-A represents DHPO with averaged mixing, while DHPO-E represents DHPO with entropy-guided mixing.
• Figure 5: Training curves of accuracy over training steps on Qwen3-1.7B-Base over seven math benchmarks. Branch-Specific Clip generally follows the same upward trend as Unified Clip while exhibiting smoother, less oscillatory updates on several datasets, consistent with reduced variance from clipping token-level and sequence-level importance ratios in separate trust regions before mixing.