QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning

TL;DR

The paper tackles the challenge of extending large reasoning models to long-context inputs under reinforcement learning, identifying suboptimal training efficiency and unstable optimization as key obstacles. It introduces QwenLong-L1, a framework that uses progressive context scaling with warm-up SFT, curriculum-guided phased RL, and difficulty-aware retrospective sampling, coupled with GRPO and DAPO under a hybrid reward scheme combining rule-based verification and LLM judgments. Evaluations on seven long-context DocQA benchmarks show substantial gains over strong baselines, with notable improvements at 14B and 32B scales and competitive performance against Claude-3.7-Sonnet-Thinking. The work provides a practical recipe for robust long-context grounding and multi-step reasoning, offering insights into SFT vs RL roles and the emergence of long-context reasoning behaviors during training.

Abstract

Recent large reasoning models (LRMs) have demonstrated strong reasoning capabilities through reinforcement learning (RL). These improvements have primarily been observed within the short-context reasoning tasks. In contrast, extending LRMs to effectively process and reason on long-context inputs via RL remains a critical unsolved challenge. To bridge this gap, we first formalize the paradigm of long-context reasoning RL, and identify key challenges in suboptimal training efficiency and unstable optimization process. To address these issues, we propose QwenLong-L1, a framework that adapts short-context LRMs to long-context scenarios via progressive context scaling. Specifically, we utilize a warm-up supervised fine-tuning (SFT) stage to establish a robust initial policy, followed by a curriculum-guided phased RL technique to stabilize the policy evolution, and enhanced with a difficulty-aware retrospective sampling strategy to incentivize the policy exploration. Experiments on seven long-context document question-answering benchmarks demonstrate that QwenLong-L1-32B outperforms flagship LRMs like OpenAI-o3-mini and Qwen3-235B-A22B, achieving performance on par with Claude-3.7-Sonnet-Thinking, demonstrating leading performance among state-of-the-art LRMs. This work advances the development of practical long-context LRMs capable of robust reasoning across information-intensive environments.

Figures (7)

• Figure 1: Overall results of across seven long-context reasoning benchmarks. Starting from R1-Distill-Qwen-32B, -32B achieves an average gain of 5.1 points, surpassing OpenAI-o3-mini, Qwen3-235B-A22B, and comparable to Claude-3.7-Sonnet-Thinking.
• Figure 2: Comparison of training dynamics between short-context and long-context reasoning RL. The long-context reasoning RL demonstrates two key challenges: suboptimal training efficiency, with slower improvements in reward score caused by more reduction in entropy, and unstable optimization process, with more fluctuations in KL divergence introduced from greater variance in longer output.
• Figure 3: Overview of , which is a novel long-context reasoning RL training framework. The proposed framework integrates group-relative RL algorithms, hybrid reward mechanisms, and progressive context scaling strategies to enable stable adaptation from short-context to long-context LRMs with robust contextual grounding and multi-step reasoning capabilities.
• Figure 4: Pass@K rates of -14B with different sample numbers across all benchmarks. We show that -14B surpasses DeepSeek-R1 with a small sampling number.
• Figure 5: Ablation studies of progressive context scaling strategy, where "Baseline" refers to the base or SFT model before RL training, "RL" refers to the naive single-stage RL, and "Phased RL" refers to the curriculum-guided phased RL. "RS" refers to the difficulty-aware retrospective sampling.