SPELL: Self-Play Reinforcement Learning for evolving Long-Context Language Models

TL;DR

SPELL presents a label-free, self-play reinforcement learning framework that enables a single LLM to autonomously evolve long-context reasoning by alternating among questioner, responder, and verifier roles.Key innovations include a self-consistent verifier, an automated curriculum with memory, and role-specific Gaussian reward shaping to maintain training stability and progressive difficulty.Across 12 open-source models and six long-context benchmarks, SPELL yields consistent improvements and often surpasses RLVR baselines, including strong gains for capable models and robustness to longer contexts.These results demonstrate SPELL’s scalability and its potential to advance long-context understanding without human supervision, while also highlighting effective strategies for stabilization and exploration.

Abstract

Progress in long-context reasoning for large language models (LLMs) has lagged behind other recent advances. This gap arises not only from the intrinsic difficulty of processing long texts, but also from the scarcity of reliable human annotations and programmatically verifiable reward signals. In this paper, we propose SPELL, a multi-role self-play reinforcement learning framework that enables scalable, label-free optimization for long-context reasoning. SPELL integrates three cyclical roles-questioner, responder, and verifier-within a single model to enable continual self-improvement. The questioner generates questions from raw documents paired with reference answers; the responder learns to solve these questions based on the documents; and the verifier evaluates semantic equivalence between the responder's output and the questioner's reference answer, producing reward signals to guide continual training. To stabilize training, we introduce an automated curriculum that gradually increases document length and a reward function that adapts question difficulty to the model's evolving capabilities. Extensive experiments on six long-context benchmarks show that SPELL consistently improves performance across diverse LLMs and outperforms equally sized models fine-tuned on large-scale annotated data. Notably, SPELL achieves an average 7.6-point gain in pass@8 on the strong reasoning model Qwen3-30B-A3B-Thinking, raising its performance ceiling and showing promise for scaling to even more capable models.

Figures (8)

• Figure 1: (Left) An overview of the framework, where a single LLM self-evolves by dynamically adopting the roles of questioner, responder, and verifier. (Right) consistently boosts performance across various models (top) and exhibits superior test-time scaling over traditional RLVR (bottom).
• Figure 2: Overview of our proposed for self-evolution of long-context reasoning. The process operates in a continuous loop that alternates between two stages: (1) Role-Specific Rollout, where a single policy model enacts three distinct roles—a questioner ($\pi_{\theta}^\text{que}$), a responder ($\pi_{\theta}^\text{res}$), and a verifier ($\pi_{\theta}^\text{ver}$)—to generate training data. (2) Unified Policy Update, where the unified policy is refined using the collected data, and the enhanced model serves as the starting point for the next rollout cycle.
• Figure 3: Test-time scaling performance (pass@k) across all benchmarks. The Qwen3-30B-A3B-Thinking model trained with shows a significantly steeper improvement as the number of samples (K) increases compared to the base model and the RLVR baseline. Notably, its pass@4 performance surpasses gemini-2.5-pro.
• Figure 4: Analysis of question difficulty (1 - pass@1) on three tasks over training steps. (Left): The full framework shows a clear upward trend in difficulty. (Middle): Without questioner updates, difficulty stagnates. (Right): Without the history memory, difficulty becomes erratic and unstable.
• Figure 5: Comparison of different reward mapping strategies. (a) Visualization of the reward functions for , AZR, and R-Zero. (b) exhibits more stable entropy dynamics during training. (c) maintains a more moderate and controlled growth in response length. (d) These factors contribute to a consistent performance improvement, ultimately leading our method to achieve the highest final pass rate.