R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training

TL;DR

This work identifies a key failure mode, Diversity Illusion, where the Solver's training signals appear diverse yet collapse into recurring underlying patterns, and proposes R-Diverse with two aligned innovations: Memory-Augmented Penalty (MAP), which uses a persistent memory bank to discourage recycling across iterations, and Skill-Aware Measurement (SAM), which evaluates diversity by the reasoning skills exercised rather than surface variation of questions.

Abstract

Self-play bootstraps LLM reasoning through an iterative Challenger-Solver loop: the Challenger is trained to generate questions that target the Solver's capabilities, and the Solver is optimized on the generated data to expand its reasoning skills. However, existing frameworks like R-Zero often exhibit non-sustained improvement, where early gains degrade as self-play continues. We identify a key failure mode, Diversity Illusion, where the Solver's training signals appear diverse yet collapse into recurring underlying patterns. It manifests as (1) Local Diversity Illusion, where diversity is enforced only within-batch, inducing cross-iteration mode cycling; and (2) Surface Diversity Illusion, where questions vary superficially but require near-identical reasoning skills. To mitigate them, we propose R-Diverse with two aligned innovations: Memory-Augmented Penalty (MAP), which uses a persistent memory bank to discourage recycling across iterations, and Skill-Aware Measurement (SAM), which evaluates diversity by the reasoning skills exercised rather than surface variation of questions. Across 10 math and general reasoning benchmarks, R-Diverse sustains gains over more iterations and consistently outperforms prior self-play methods. Code is available at https://github.com/Gengsheng-Li/R-Diverse.

Figures (4)

• Figure 1: Overview of Diversity Illusion and the R-Diverse framework. (a) Despite a decreasing repetition penalty, cross-iteration and intra-iteration repetition increase, revealing a mismatch between what is penalized and what the Solver is trained on. (b) Existing methods exhibit Local Diversity Illusion and Surface Diversity Illusion. (c) R-Diverse resolves these failures with MAP to enforce global, history-aware exploration and SAM to identify repetitions at the level of underlying reasoning skills. (d) Consequently, R-Diverse sustains improvement over five iterations (52.59), avoiding the collapse observed in R-Zero.
• Figure 2: The R-Diverse framework. Top (Challenger training): the Challenger proposes questions $\{q_i\}$ to maximize the uncertainty reward $R_{\text{uncertainty}}$. For each $q_i$, the Solver produces multiple solutions $\{a_{i,1},a_{i,2},\ldots,a_{i,m}\}$ and a pseudo-label $\hat{y}_i$ via majority voting. SAM maps $q_i$ to canonical solver code $c_i$ (via Qwen2.5-Coder-7B) and a semantic embedding $e_i$, which are used to compute both the within-iteration repetition penalty $P_{\text{rep}}$ and the memory-augmented penalty $P_{\text{MAP}}$. Middle (Memory update): the memory bank $\mathcal{M}$ stores historical tuples $(q_i,e_i,\hat{y}_i)$ across iterations. Bottom (Solver training): to mitigate distribution shift, the Solver is trained on current questions augmented with samples recalled from $\mathcal{M}$, using a matching reward $R_{\text{match}}$.
• Figure 3: Performance across iterations on different metrics and model scales. R-Diverse achieves monotonic improvement across all settings, while R-Zero collapses after iteration 3-4. Stars indicate peak performance.
• Figure 4: Multi-dimensional diversity analysis. (a-b) Cross-Iteration Repetition: R-Zero exhibits increasing historical recycling (confirmed by both SAM embedding similarity and the LLM judge), while R-Diverse reduces it. (c-d) Intra-Iteration Diversity: R-Diverse maintains lower repetition and higher distribution spread. (e) Policy Dynamics: R-Diverse recovers Challenger entropy, avoiding the low-entropy collapse observed in R-Zero. ($\uparrow$: higher is better; $\downarrow$: lower is better.)