LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts

TL;DR

LoongRL tackles the challenge of enabling reinforcement learning to perform advanced reasoning over very long contexts. It introduces KeyChain, a data synthesis method that embeds UUID-based chains into long-context inputs to force explicit chain tracing, retrieval, and reasoning, then trains with Group Relative Policy Optimization using a rule-based verifier. The results show substantial improvements on long-context multi-hop QA, with LoongRL-14B reaching a 74.2 score that rivals larger frontier models, and the approach generalizes from $16K$ to $128K$ contexts while preserving short-context capabilities. This work demonstrates that high-quality long-context reasoning patterns can emerge from data-driven RL and provides reproducible data, prompts, and training procedures to enable scaling and adoption.

Abstract

Reasoning over long contexts is essential for large language models. While reinforcement learning (RL) enhances short-context reasoning by inducing "Aha" moments in chain-of-thought, the advanced thinking patterns required for long-context reasoning remain largely unexplored, and high-difficulty RL data are scarce. In this paper, we introduce LoongRL, a data-driven RL method for advanced long-context reasoning. Central to LoongRL is KeyChain, a synthesis approach that transforms short multi-hop QA into high-difficulty long-context tasks by inserting UUID chains that hide the true question among large collections of distracting documents. Solving these tasks requires the model to trace the correct chain step-by-step, identify the true question, retrieve relevant facts and reason over them to answer correctly. RL training on KeyChain data induces an emergent plan-retrieve-reason-recheck reasoning pattern that generalizes far beyond training length. Models trained at 16K effectively solve 128K tasks without prohibitive full-length RL rollout costs. On Qwen2.5-7B and 14B, LoongRL substantially improves long-context multi-hop QA accuracy by +23.5% and +21.1% absolute gains. The resulting LoongRL-14B reaches a score of 74.2, rivaling much larger frontier models such as o3-mini (74.5) and DeepSeek-R1 (74.9). It also improves long-context retrieval, passes all 128K needle-in-a-haystack stress tests, and preserves short-context reasoning capabilities.

Figures (9)

• Figure 1: Model trajectories on long-context multi-hop QA with and without KeyChain RL data. (a) With KeyChain data, model exhibits an emergent plan–retrieve–reason–recheck thinking pattern, improving reasoning reliability and can generalize to longer contexts. (b) Without KeyChain data, reasoning and retrieval are entangled, the model often lacks an explicit planning step and does not deeply reason over retrieved information, frequently leading to errors. Reasoning steps are marked in blue and retrieval steps in orange.
• Figure 2: Overview of our KeyChain data construction.
• Figure 3: Needle in a Haystack retrieval across document depths. The base Qwen2.5-7B-Instruct does not fully pass the benchmark, whereas LoongRL-7B achieves perfect 100% retrieval accuracy.
• Figure 4: Long-context reasoning accuracy and training response lengths throughout RL training.
• Figure 5: System prompt used during training