Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM

Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, Shimiao Jiang, Shiqi Kuang, Shouyu Yin, Chaohang Wen, Haotian Zhang, Bin Chen, Bing Yu

TL;DR

SRPO tackles cross-domain reinforcement learning for large language models by addressing GRPO limitations through a two-stage training regime and history resampling. The method first builds strong mathematical reasoning, then integrates coding skills atop the developed reasoning foundation, supported by a data curation pipeline and a rule-based reward design. Empirical results show SRPO outperforms prior multi-domain methods on AIME24 and LiveCodeBench using roughly one-tenth the training steps of the R1-Zero baseline, illustrating improved sample efficiency and cross-domain transfer. The work provides insights into thinking patterns and progression of problem-solving strategies in large-scale RL for math and code tasks.

Abstract

Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks.

SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM

TL;DR

SRPO tackles cross-domain reinforcement learning for large language models by addressing GRPO limitations through a two-stage training regime and history resampling. The method first builds strong mathematical reasoning, then integrates coding skills atop the developed reasoning foundation, supported by a data curation pipeline and a rule-based reward design. Empirical results show SRPO outperforms prior multi-domain methods on AIME24 and LiveCodeBench using roughly one-tenth the training steps of the R1-Zero baseline, illustrating improved sample efficiency and cross-domain transfer. The work provides insights into thinking patterns and progression of problem-solving strategies in large-scale RL for math and code tasks.

Abstract

Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks.

Paper Structure

This paper contains 22 sections, 6 equations, 18 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. GRPO
  4. Training Template
  5. Approach
  6. Challenges with Vanilla GRPO Implementation
  7. Two Stage Training
  8. Comparative Analysis of Training Strategies
  9. Mixed Training
  10. Math-Only Training
  11. Code-Only Training
  12. Staged Training
  13. History Resampling (HR)
  14. Data Curation
  15. Reward Design
  16. ...and 7 more sections

Figures (18)

  • Figure 1: SRPO achieves superior results with only 10% of DeepSeek's training steps. The values shown are pass@1 scores, averaged over 32 samples per question.
  • Figure 2: Math and Code training templates.
  • Figure 3: Response length of different training data strategies.
  • Figure 4: Case of Mixed Training.
  • Figure 5: Case of Math-Only Training.
  • ...and 13 more figures