MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning

TL;DR

This paper proposes Mass-Adaptive Soft Policy Optimization (MASPO), a unified framework designed to harmonize these three dimensions of RLVR, which integrates a differentiable soft Gaussian gating to maximize gradient utility, a mass-adaptive limiter to balance exploration across the probability spectrum, and an asymmetric risk controller to align update magnitudes with signal confidence.

Abstract

Existing Reinforcement Learning with Verifiable Rewards (RLVR) algorithms, such as GRPO, rely on rigid, uniform, and symmetric trust region mechanisms that are fundamentally misaligned with the complex optimization dynamics of Large Language Models (LLMs). In this paper, we identify three critical challenges in these methods: (1) inefficient gradient utilization caused by the binary cutoff of hard clipping, (2) insensitive probability mass arising from uniform ratio constraints that ignore the token distribution, and (3) asymmetric signal reliability stemming from the disparate credit assignment ambiguity between positive and negative samples. To bridge these gaps, we propose Mass-Adaptive Soft Policy Optimization (MASPO), a unified framework designed to harmonize these three dimensions. MASPO integrates a differentiable soft Gaussian gating to maximize gradient utility, a mass-adaptive limiter to balance exploration across the probability spectrum, and an asymmetric risk controller to align update magnitudes with signal confidence. Extensive evaluations demonstrate that MASPO serves as a robust, all-in-one RLVR solution, significantly outperforming strong baselines. Our code is available at: https://anonymous.4open.science/r/ma1/README.md.

Figures (7)

• Figure 1: MASPO resolves three core limitations of GRPO: (1) inefficient gradient utilization caused by hard boundaries that indiscriminately discard valid gradients; (2) insensitive probability mass, where uniform clipping overlooks the head/tail mass disparity; and (3) asymmetric signal reliability, which neglects the inherent noise differences between verified positive and ambiguous negative signals.
• Figure 2: Overview of the MASPO framework. The architecture integrates a Mass-Adaptive Limiter to scale constraints inversely with token probability and an Asymmetric Risk Controller to modulate update magnitude based on advantage signals, unified via a differentiable Soft Gaussian Gating mechanism.
• Figure 3: Evolution of Training Dynamics and Performance across Model Scales. Top row: 1.5B model; Bottom row: 7B model. MASPO demonstrates superior convergence, achieving higher performance ceilings.
• Figure 4: Hyperparameter sensitivity and scalability analysis. Top: MASPO demonstrates robustness across a range of $\alpha$ and $\beta$ values, outperforming the GRPO baseline. Bottom: Training dynamics on the larger 14B model, demonstrating MASPO's scalability by maintaining higher entropy and superior performance compared to GRPO.
• Figure 5: Supplementary training dynamics for additional baselines (Clip Higher, Advantage Reweighting, etc.).