Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computation

Yutong Chen, Jiandong Gao, Ji Wu

TL;DR

CoScale-RL tackles instability and limited ability boundaries in post-training LRMs by co-scaling data (more solutions per problem) and computation (larger Rollout N) within an iterative SFT/RL loop, guided by a theoretical framework and reinforced with Re-distillation for efficient model merging. The approach uses solvable/unsolvable/solved problem pools to manage progression and avoids catastrophic forgetting via final retraining, achieving a 3.76x average accuracy boost across math benchmarks and breaching prior ability boundaries on a 0.5B LM. A key contribution is the quadratic relationship between $\eta/N$ and computational efficiency, yielding practical guidance for stable RL and efficiency gains, plus evidence that data scaling in RL, rather than algorithmic tweaks alone, drives most performance gains. The results demonstrate a scalable post-training direction that reduces reliance on extensive SFT data and generalizes to larger LMs, offering a principled path to stronger LRM reasoning in resource-constrained settings.

Abstract

Training Large Reasoning Model (LRM) is usually unstable and unpredictable, especially on hard problems or weak foundation models. We found that the current post-training scaling strategy can still improve on these cases. We propose CoScale-RL, a novel scaling strategy with better data and computational efficiency. We first scale up solutions to make problems solvable. The core idea is to collect multiple solutions for each problem, rather than simply enlarging the dataset. Then, we scale up rollout computation to stabilize Reinforcement Learning. We further leverage a model merge technique called Re-distillation to sustain or even improve computational efficiency when scaling up. Our method significantly improves data and computational efficiency, with an average 3.76$\times$ accuracy improvement on four benchmarks. CoScale-RL is able to improve an LRM's ability boundary without an extensive SFT dataset. Our method provides a new scaling direction to further improve LRM's reasoning ability.

CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computation

TL;DR

CoScale-RL tackles instability and limited ability boundaries in post-training LRMs by co-scaling data (more solutions per problem) and computation (larger Rollout N) within an iterative SFT/RL loop, guided by a theoretical framework and reinforced with Re-distillation for efficient model merging. The approach uses solvable/unsolvable/solved problem pools to manage progression and avoids catastrophic forgetting via final retraining, achieving a 3.76x average accuracy boost across math benchmarks and breaching prior ability boundaries on a 0.5B LM. A key contribution is the quadratic relationship between and computational efficiency, yielding practical guidance for stable RL and efficiency gains, plus evidence that data scaling in RL, rather than algorithmic tweaks alone, drives most performance gains. The results demonstrate a scalable post-training direction that reduces reliance on extensive SFT data and generalizes to larger LMs, offering a principled path to stronger LRM reasoning in resource-constrained settings.

Abstract

Training Large Reasoning Model (LRM) is usually unstable and unpredictable, especially on hard problems or weak foundation models. We found that the current post-training scaling strategy can still improve on these cases. We propose CoScale-RL, a novel scaling strategy with better data and computational efficiency. We first scale up solutions to make problems solvable. The core idea is to collect multiple solutions for each problem, rather than simply enlarging the dataset. Then, we scale up rollout computation to stabilize Reinforcement Learning. We further leverage a model merge technique called Re-distillation to sustain or even improve computational efficiency when scaling up. Our method significantly improves data and computational efficiency, with an average 3.76 accuracy improvement on four benchmarks. CoScale-RL is able to improve an LRM's ability boundary without an extensive SFT dataset. Our method provides a new scaling direction to further improve LRM's reasoning ability.
Paper Structure (39 sections, 2 theorems, 21 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 39 sections, 2 theorems, 21 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. RL Ability Boundary
  4. Scaling Law of Reinforcement Learning
  5. Computational Efficiency
  6. Methods
  7. CoScale-RL : Co-scaling Data and Computation
  8. Theoretical Analysis for Efficient Computational Scaling
  9. Optimal $\eta/N$ to Maximize Efficiency
  10. The Effect of Problem Difficulty
  11. Takeaways to Maximize Computational Efficiency
  12. Experiments
  13. Experimental Settings
  14. Baseline Settings
  15. Results
  16. ...and 24 more sections

Key Result

Theorem 3.1

Under some assumptions, the computational efficiency of both SGD and Adam Optimizer can be described as: where $\mu_p$ and $\mu_n$ are not related to $\eta$ or $N$.

Figures (7)

  • Figure 1: CoScale-RL consistently outperforms baselines on multiple benchmarks. Compared with the Pretrained (0.5B), our method has a 3.76$\times$ average improvement on four math reasoning benchmarks. Refer to \ref{['sec:exp-large-scale']} for details. Error Bar: 95% CI.
  • Figure 2: Three core techniques in CoScale-RL. The definition of Solvable/Unsolvable/Solved and other details can be found in \ref{['sec:method']}.
  • Figure 3: Overview of our method. Co-scaling Strategy (Left): We simultaneously scale up Solution per Problem in SFT and Rollout N in RL. After entering a new iteration, we gather more solutions for unsolvable problems. Once a problem becomes solvable, we scale up computation until RL solves this problem. Data Management (Middle): All problems are labeled as unsolvable at start. After evaluation, we label those problems as solvable with enough accuracy. We only collect solvable problems for RL. Training Pipeline (Right): We finetune the initial model with a few problems to switch output mode. Then we follow the same iterative process. In the final iteration, we retrain the model from the initial model to avoid catastrophic forgetting.
  • Figure 4: 1.5B LLM can solve an AIME level problem after being finetuned on 50 solutions.Upper: Training pipeline. Lower (Left): RL on only one problem. We found a stable improvement in reward and a steady 10k level response length. Lower (Right): RL on 8 augmented problems. RLed model succeeded in solving unseen problems after training for 80 steps. Shadow Area: 95% CI.
  • Figure 5: Some problems need more solutions to be solvable. Top: Long-tailed distribution of minimal required solutions per problem. Bottom: Scaling up the solution per problem improves both Pass@1 and Pass@16, which is more effective than scaling up the dataset size. This experiment is based on Qwen2.5-1.5B Instruct. Error bar: 95% CI.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 3.1
  • Lemma 1.1