Adaptive-Boundary-Clipping GRPO: Ensuring Bounded Ratios for Stable and Generalizable Training

TL;DR

This work tackles instability and limited generalization in Group Relative Policy Optimization (GRPO) caused by sequence-level advantages and asymmetric clipping. It introduces Adaptive-Boundary-Clipping GRPO (ABC-GRPO), which uses four unconditional clipping boundaries across the four quadrants of the ($r$, $\hat{A}$) space, clipping before applying the advantage to bound updates in all cases. The authors prove a bounded-gradient guarantee and demonstrate empirical gains on mathematical reasoning tasks with Qwen3 LLMs, including higher final performance, monotonic improvements in Pass@k, and substantially higher entropy to preserve exploration. The method achieves strong, scalable improvements with minimal code changes and offers a principled direction for stable RLHF training of large language models.

Abstract

Group Relative Policy Optimization (GRPO) has emerged as a popular algorithm for reinforcement learning with large language models (LLMs). However, upon analyzing its clipping mechanism, we argue that it is suboptimal in certain scenarios. With appropriate modifications, GRPO can be significantly enhanced to improve both flexibility and generalization. To this end, we propose Adaptive-Boundary-Clipping GRPO (ABC-GRPO), an asymmetric and adaptive refinement of the original GRPO framework. We demonstrate that ABC-GRPO achieves superior performance over standard GRPO on mathematical reasoning tasks using the Qwen3 LLMs. Moreover, ABC-GRPO maintains substantially higher entropy throughout training, thereby preserving the model's exploration capacity and mitigating premature convergence. The implementation code is available online to ease reproducibility https://github.com/chi2liu/ABC-GRPO.

Figures (4)

• Figure 1: The credit assignment problem in GRPO. Two responses differ only in the final token, yet GRPO assigns opposite advantage signs to the identical first 8 tokens.
• Figure 2: Four-quadrant analysis of GRPO's clipping in $(r, \hat{A})$ space. Q1 and Q3 are protected by clipping. Q2 is a mild blind spot ($r < 1$ bounds magnitude). Q4 is particularly problematic: when $\hat{A} < 0$ and $r \gg 1$, GRPO's $\max(r, 1 - \varepsilon)$ provides no protection, enabling unbounded policy updates.
• Figure 3: Closing GRPO's blind spots. Left: GRPO's inherited sign-dependent clipping leaves two quadrants unprotected (red). Right: ABC-GRPO applies unconditional clipping with four independent parameters ($\varepsilon_1, \varepsilon_2, \varepsilon_3, \varepsilon_4$), closing all blind spots. For $\hat{A} > 0$, $\varepsilon_1$ and $\varepsilon_2$ control upper and lower bounds; for $\hat{A} < 0$, $\varepsilon_3$ and $\varepsilon_4$ provide the bounds.
• Figure 4: Training dynamics for GRPO vs ABC-GRPO (Qwen3-4B). Top row shows performance metrics: (a) Avg@64 progression, (b) Pass@64 evolution showing GRPO degradation, (c) Pass@32 curves. Bottom row shows diagnostic metrics: (d) Entropy preservation, (e) Training reward curves, (f) Clipping distribution by quadrant, validating Q4 as the critical blind spot (41.4% of events).