Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning

Hanxu Hu, Yuxuan Wang, Maggie Huan, Jannis Vamvas, Yinya Huang, Zhijiang Guo, Rico Sennrich

Abstract

Reinforcement learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for eliciting reasoning capabilities in large language models, particularly in mathematics and coding. While recent efforts have extended this paradigm to broader general scientific (STEM) domains, the complex interplay between supervised fine-tuning (SFT) and RL in these contexts remains underexplored. In this paper, we conduct controlled experiments revealing a critical challenge: for general STEM domains, RL applied directly to base models is highly sample-inefficient and is consistently surpassed by supervised fine-tuning (SFT) on moderate-quality responses. Yet sequential SFT followed by RL can further improve performance, suggesting that the two stages play complementary roles, and that how training data is allocated between them matters. Therefore, we propose DeReason, a difficulty-based data decoupling strategy for general reasoning. DeReason partitions training data by reasoning intensity estimated via LLM-based scoring into reasoning-intensive and non-reasoning-intensive subsets. It allocates broad-coverage, non-reasoning-intensive problems to SFT to establish foundational domain knowledge, and reserves a focused subset of difficult problems for RL to cultivate complex reasoning. We demonstrate that this principled decoupling yields better performance than randomly splitting the data for sequential SFT and RL. Extensive experiments on general STEM and mathematical benchmarks demonstrate that our decoupled curriculum training significantly outperforms SFT-only, RL-only, and random-split baselines. Our work provides a systematic study of the interplay between SFT and RL for general reasoning, offering a highly effective and generalized post-training recipe.

DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning

Abstract

Reinforcement learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for eliciting reasoning capabilities in large language models, particularly in mathematics and coding. While recent efforts have extended this paradigm to broader general scientific (STEM) domains, the complex interplay between supervised fine-tuning (SFT) and RL in these contexts remains underexplored. In this paper, we conduct controlled experiments revealing a critical challenge: for general STEM domains, RL applied directly to base models is highly sample-inefficient and is consistently surpassed by supervised fine-tuning (SFT) on moderate-quality responses. Yet sequential SFT followed by RL can further improve performance, suggesting that the two stages play complementary roles, and that how training data is allocated between them matters. Therefore, we propose DeReason, a difficulty-based data decoupling strategy for general reasoning. DeReason partitions training data by reasoning intensity estimated via LLM-based scoring into reasoning-intensive and non-reasoning-intensive subsets. It allocates broad-coverage, non-reasoning-intensive problems to SFT to establish foundational domain knowledge, and reserves a focused subset of difficult problems for RL to cultivate complex reasoning. We demonstrate that this principled decoupling yields better performance than randomly splitting the data for sequential SFT and RL. Extensive experiments on general STEM and mathematical benchmarks demonstrate that our decoupled curriculum training significantly outperforms SFT-only, RL-only, and random-split baselines. Our work provides a systematic study of the interplay between SFT and RL for general reasoning, offering a highly effective and generalized post-training recipe.
Paper Structure (26 sections, 2 equations, 6 figures, 3 tables)

This paper contains 26 sections, 2 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. RL vs. SFT under Controlled Comparison.
  4. Preliminaries
  5. Supervised Fine-Tuning (SFT).
  6. Reinforcement Learning with GRPO.
  7. Verification for General Reasoning
  8. DeReason: Difficulty-Based Data Decoupling
  9. Overall Pipeline.
  10. Difficulty Estimation.
  11. LLM-based Scoring.
  12. Experiments
  13. Training
  14. Evaluation
  15. Baselines
  16. ...and 11 more sections

Figures (6)

  • Figure 1: Scaling behavior of RL and SFT on both general STEM and math domains, trained on the same set of problems. In the general domain, we report pass@1 averaged across 8 runs on GPQA-Diamond. In the math domain, we report the same metric averaged across AIME24, AIME25, and MATH500. In both settings, SFT with moderate-quality model responses outperforms RL as training data increases.
  • Figure 2: The category distribution across difficulty scores.
  • Figure 3: Reward score during RL training with data in different difficulty score (from 1 to 5) from Webinstruct-verified. The left sub figure is starting from SFT checkpoint, right sub figure is starting from base checkpoint
  • Figure 4: Response length during RL training with data in different difficulty scores.
  • Figure 5: Actor's entropy during RL training with different initial checkpoint.
  • ...and 1 more figures