Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

TL;DR

<3-5 sentence high-level summary> This survey provides a lifecycle-centered review of reinforcement learning for large language models, with a particular emphasis on Reinforcement Learning with Verifiable Rewards (RLVR) as a mechanism to obtain reliable, testable improvements in reasoning and alignment. It systematically covers theory (policy vs value learning), RL across pre-training, alignment, and reasoning phases, and the growing role of multimodal and agentic applications. The work also compiles datasets, benchmarks, and open-source frameworks, and discusses key challenges—scalability, reward design, and theoretical understanding—and forward-looking trends toward richer reward signals and safer, more capable RL-enabled systems. Overall, it aims to guide researchers and practitioners toward more intelligent, generalizable, and safe RL-enhanced language systems across the full lifecycle.

Abstract

In recent years, training methods centered on Reinforcement Learning (RL) have markedly enhanced the reasoning and alignment performance of Large Language Models (LLMs), particularly in understanding human intents, following user instructions, and bolstering inferential strength. Although existing surveys offer overviews of RL augmented LLMs, their scope is often limited, failing to provide a comprehensive summary of how RL operates across the full lifecycle of LLMs. We systematically review the theoretical and practical advancements whereby RL empowers LLMs, especially Reinforcement Learning with Verifiable Rewards (RLVR). First, we briefly introduce the basic theory of RL. Second, we thoroughly detail application strategies for RL across various phases of the LLM lifecycle, including pre-training, alignment fine-tuning, and reinforced reasoning. In particular, we emphasize that RL methods in the reinforced reasoning phase serve as a pivotal driving force for advancing model reasoning to its limits. Next, we collate existing datasets and evaluation benchmarks currently used for RL fine-tuning, spanning human-annotated datasets, AI-assisted preference data, and program-verification-style corpora. Subsequently, we review the mainstream open-source tools and training frameworks available, providing clear practical references for subsequent research. Finally, we analyse the future challenges and trends in the field of RL-enhanced LLMs. This survey aims to present researchers and practitioners with the latest developments and frontier trends at the intersection of RL and LLMs, with the goal of fostering the evolution of LLMs that are more intelligent, generalizable, and secure.

Figures (3)

• Figure 1: A taxonomy of RL enhanced LLMs. This figure presents a taxonomy of the key stages and resources involved in creating RL-enhanced LLMs, organized into five branches: pre-training, alignment, RLVR, datasets & benchmarks, and open-source frameworks. The taxonomy clarifies the interconnections between stages, serving as a roadmap for understanding methodological advancements and resources discussed in the survey.
• Figure 2: Key components in RL-enhanced LLMs. This figure illustrates the key components and their interactions within the lifecycle of RL-enhanced LLMs. Driven by RL frameworks and toolkits, RL algorithms participate in the pre-training, alignment, and reasoning enhancement training of LLMs, and are validated through test benchmarks.
• Figure 3: Technical architecture of the RLVR methods. It depicts the overall workflow of the RLVR and expands on the design methods for the reward model, off-policy assistance, reward filtering, sampling and reasoning strategies, Agent RL, and reward update hierarchy.