100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models

TL;DR

This survey analyzes open-source replication efforts of DeepSeek-R1, detailing supervised fine-tuning (SFT) and reinforcement learning from verifiable rewards (RLVR) pipelines, data curation, and training practices. It maps the landscape of SFT datasets, their curation, verification, and cross-dataset relationships, and compares SFT outcomes across starting checkpoints and model scales. It then dissects RLVR components, algorithms (PPO, GRPO, REINFORCE variants), reward schemes, data strategies, and their impact on long-form reasoning across math, coding, and general tasks. Finally, it discusses broader directions—reward modeling, preference optimization, generalizability, safety, and multimodal/multilingual extensions—to guide future research toward robust, scalable, and safe reasoning-language models.

Abstract

The recent development of reasoning language models (RLMs) represents a novel evolution in large language models. In particular, the recent release of DeepSeek-R1 has generated widespread social impact and sparked enthusiasm in the research community for exploring the explicit reasoning paradigm of language models. However, the implementation details of the released models have not been fully open-sourced by DeepSeek, including DeepSeek-R1-Zero, DeepSeek-R1, and the distilled small models. As a result, many replication studies have emerged aiming to reproduce the strong performance achieved by DeepSeek-R1, reaching comparable performance through similar training procedures and fully open-source data resources. These works have investigated feasible strategies for supervised fine-tuning (SFT) and reinforcement learning from verifiable rewards (RLVR), focusing on data preparation and method design, yielding various valuable insights. In this report, we provide a summary of recent replication studies to inspire future research. We primarily focus on SFT and RLVR as two main directions, introducing the details for data construction, method design and training procedure of current replication studies. Moreover, we conclude key findings from the implementation details and experimental results reported by these studies, anticipating to inspire future research. We also discuss additional techniques of enhancing RLMs, highlighting the potential of expanding the application scope of these models, and discussing the challenges in development. By this survey, we aim to help researchers and developers of RLMs stay updated with the latest advancements, and seek to inspire new ideas to further enhance RLMs.

Figures (3)

• Figure 1: Taxonomy of training methods of reasoning models.
• Figure 2: Token length distributions for the aforementioned SFT datasets. The x-axis is truncated at 20,000 tokens, as examples exceeding this length are rare.
• Figure 3: An illustration of cross-referenced dataset sources for popular math reasoning datasets. Arrows point from source datasets to target datasets that incorporate some of their data. The figure does not reflect dataset sizes, nor does it imply that a target dataset includes all data from its source, or only data from the source(s) indicated by the arrows. Datasets highlighted in lilac contain Chain-of-Thought traces extracted from DeepSeek-R1.