A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law

TL;DR

The survey investigates slow-thinking-based reasoning in LLMs, arguing that deliberate, multi-step reasoning guided by inference-time scaling and reinforcement learning forms a promising path toward robust AI. It synthesizes insights from over 100 studies, proposing a taxonomy across test-time scaling, reinforced learning, and slow-thinking frameworks, with emphasis on RL paradigms (PPO, DPO, GRPO) and verification-enabled reasoning. Key contributions include a comprehensive review of reward models (ORM/PRM, rule-based, model-based), distillation (model and data), search strategies (Greedy, Beam, Best-of-N, MCTS), and self-training, along with systematic coverage of long CoT, hierarchical reasoning, and hybrid thinking. The paper highlights challenges in RL stability, reward design, and generalization, and outlines directions for multi-modal reasoning, human-in-the-loop refinement, and self-improving RL frameworks to broaden real-world impact.

Abstract

This survey explores recent advancements in reasoning large language models (LLMs) designed to mimic "slow thinking" - a reasoning process inspired by human cognition, as described in Kahneman's Thinking, Fast and Slow. These models, like OpenAI's o1, focus on scaling computational resources dynamically during complex tasks, such as math reasoning, visual reasoning, medical diagnosis, and multi-agent debates. We present the development of reasoning LLMs and list their key technologies. By synthesizing over 100 studies, it charts a path toward LLMs that combine human-like deep thinking with scalable efficiency for reasoning. The review breaks down methods into three categories: (1) test-time scaling dynamically adjusts computation based on task complexity via search and sampling, dynamic verification; (2) reinforced learning refines decision-making through iterative improvement leveraging policy networks, reward models, and self-evolution strategies; and (3) slow-thinking frameworks (e.g., long CoT, hierarchical processes) that structure problem-solving with manageable steps. The survey highlights the challenges and further directions of this domain. Understanding and advancing the reasoning abilities of LLMs is crucial for unlocking their full potential in real-world applications, from scientific discovery to decision support systems.

Figures (4)

• Figure 1: The timeline of main reasoning LLMs.
• Figure 2: Roadmap to Reasoning LLM.
• Figure 3: The search algorithms for test-time scaling
• Figure 4: The reinforcement learning framework and reward model