The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward

TL;DR

The paper addresses diversity collapse in RLVR, where improved Pass@1 often comes at the expense of Pass@k and generalization. It reframes the KL-divergence term as a rehearsal mechanism by adopting mass-covering $f$-divergences (notably $D_{ ext{forward-KL}}$ and $D_{ ext{JS}}$) that reference the initial policy, yielding Diversity-Preserving Hybrid RL (DPH-RL). DP HRL partitions data into near-perfect and exploration sets, using a generator-based online scheme with two losses: an exploration objective without KL on $\mathcal{D}_{exp}$ and an $f$-divergence penalty on $\mathcal{D}_{pef}$, leading to improved Pass@1 and Pass@k in-domain and out-of-domain. The authors provide an enhanced monotonic-improvement guarantee and demonstrate through SQL and mathematical reasoning tasks that DPH-RL achieves better diversity, stability, and training efficiency than reverse-KL and prior baselines. This work offers a principled, scalable axis for improving RLVR by leveraging the properties of $f$-divergences to preserve diverse reasoning capabilities in large language models.

Abstract

A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance (Pass@k) despite improvements in single-attempt accuracy (Pass@1). This is often accompanied by catastrophic forgetting, where models lose previously acquired skills. While various methods have been proposed, the choice and function of the divergence term have been surprisingly unexamined as a proactive solution. We argue that standard RLVR objectives -- both those using the mode-seeking reverse KL-divergence and those forgoing a divergence term entirely -- lack a crucial mechanism for knowledge retention. The reverse-KL actively accelerates this decay by narrowing the policy, while its absence provides no safeguard against the model drifting from its diverse knowledge base. We propose a fundamental shift in perspective: using the divergence term itself as the solution. Our framework, Diversity-Preserving Hybrid RL (DPH-RL), leverages mass-covering f-divergences (like forward-KL and JS-divergence) to function as a rehearsal mechanism. By continuously referencing the initial policy, this approach forces the model to maintain broad solution coverage. Extensive experiments on math and SQL generation demonstrate that DPH-RL not only resolves the Pass@k degradation but improves both Pass@1 and Pass@k in- and out-of-domain. Additionally, DPH-RL is more training-efficient because it computes f-divergence using generator functions, requiring only sampling from the initial policy and no online reference model. Our work highlights a crucial, overlooked axis for improving RLVR, demonstrating that the proper selection of a divergence measure is a powerful tool for building more general and diverse reasoning models.

Figures (6)

• Figure 1: The left panel evaluates the performance gap in Pass@k between the RL-trained model and the Base Model across test sets with varying degrees of divergence from the training data. The right panel visualizes the distributions of reverse-KL and forward-KL.
• Figure 2: On the left, we construct a base model that outputs multiple solution styles for SQL problems. This model is then used for reinforcement learning training. We calculated the probability of the number of times the model outputted different styles across 32 samples.
• Figure 3: $\eta$ vs. Greedy
• Figure 4: Exploring the differences between base model and RL-Tuned models in Llama.
• Figure 5: Llama Pass@8 progress: Algorithms vs. Training Steps

Theorems & Definitions (6)

• Theorem 1: Enhanced Monotonic Improvement
• Lemma 1
• proof
• Lemma 2: schulman2015trust
• Lemma 3: Lemma 4 in kang2018policy
• proof