Rethinking GSPO: The Perplexity-Entropy Equivalence

TL;DR

This paper connects GSPO's sequence-level importance weights to information-theoretic quantities by proving $s(\theta) = (\pi_\theta(y|x)/\pi_{\theta_{\text{old}}}(y|x))^{1/|y|} = \text{PPL}_{\theta_{\text{old}}}(y|x)/\text{PPL}_\theta(y|x) = \exp(\Delta H)$, thereby reframing policy updates as perplexity and entropy changes. It shows that the perplexity-based formulation yields log-domain variance reduction through geometric averaging, and explains MoE stability, long-sequence advantages, and hyperparameter intuitions. Theoretical consequences are supported by experiments on mathematical reasoning tasks, which validate the equivalence with mean errors below $0.05\%$, observe $O(1/|y|)$ variance scaling in the log domain, and demonstrate substantial perplexity improvements ($75.2\%$) and entropy reductions ($81.6\%$). This information-theoretic perspective links language-model metrics to policy optimization, offering principled guidance for designing stable, scalable sequence-level RL methods that leverage perplexity dynamics.

Abstract

We provide a new perspective on GSPO's length-normalized importance ratios by establishing their connection to information-theoretic quantities. We show that GSPO's sequence-level weight $s(θ) = (π_θ/π_{θ_{\text{old}}})^{1/|y|}$ can be equivalently expressed as the inverse perplexity ratio $\text{PPL}_{θ_{\text{old}}}/\text{PPL}_θ$ and as the exponential cross-entropy change $\exp(ΔH)$. While the perplexity-entropy relationship follows from standard definitions, this observation provides a useful lens for understanding GSPO: the algorithm weights policy gradient updates by perplexity ratios, offering an information-theoretic interpretation of the importance weights. This perspective helps explain GSPO's empirical properties, including log-domain variance reduction through geometric averaging and stability in training mixture-of-experts models. We validate the mathematical equivalences and variance predictions through controlled experiments on mathematical reasoning tasks.

Figures (2)

• Figure 1: Information-Theoretic View of GSPO. Length normalization $(\cdot)^{1/|y|}$ transforms token probabilities through sequence probability and cross-entropy to perplexity. Our observation: taking ratios of length-normalized sequence probabilities is equivalent to computing perplexity ratios, which equals exponential cross-entropy changes. This connection provides an information-theoretic interpretation of GSPO's importance weights.
• Figure 2: Experimental Validation of Theoretical Framework. (a) Core equivalence $s(\theta) = \text{PPL}_{\text{ratio}} = \exp(\Delta H)$ verified with mean error $<0.05\%$. (b) Log-domain variance reduction matches theoretical prediction within $3.6\times$ factor. (c) Perplexity improves by $75.2\%$ from 4.59 to 1.14. All results from 1140 training steps on mathematical reasoning tasks with $\epsilon \in [3\times10^{-4}, 4\times10^{-4}]$.

Theorems & Definitions (10)

• Theorem 3.1: Perplexity-Ratio Equivalence
• proof
• Theorem 3.2: Entropy Reduction Characterization
• proof
• Theorem 4.1: Log-Domain Variance Reduction
• Remark 4.2
• Theorem 4.3: Gradient Weighting by Information Gain
• Remark 4.4
• proof
• proof