Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning

Caijun Xu, Changyi Xiao, Zhongyuan Peng, Xinrun Wang, Yixin Cao

TL;DR

SCALER tackles waning RL rewards during long-horizon reasoning training by introducing a scalable environment synthesis pipeline and adaptive multi-environment RL. It converts real programming problems into verifiable reasoning environments with controllable difficulty and unbounded instances, and couples this with an online difficulty controller and environment curation to keep training on the model's capability frontier while preserving diversity. Empirical results show SCALER outperforms dataset-based RL baselines across five reasoning benchmarks and exhibits more stable, long-horizon training dynamics. This framework provides a scalable platform to study how environment properties influence RL for reasoning and demonstrates the importance of balancing difficulty with diversity during continual learning.

Abstract

Reinforcement learning (RL) offers a principled way to enhance the reasoning capabilities of large language models, yet its effectiveness hinges on training signals that remain informative as models evolve. In practice, RL progress often slows when task difficulty becomes poorly aligned with model capability, or when training is dominated by a narrow set of recurring problem patterns. To jointly address these issues, we propose SCALER (Synthetic sCalable Adaptive Learning Environment for Reasoning), a framework that sustains effective learning signals through adaptive environment design. SCALER introduces a scalable synthesis pipeline that converts real-world programming problems into verifiable reasoning environments with controllable difficulty and unbounded instance generation, enabling RL training beyond finite datasets while preserving strong correctness guarantees. Building on this, SCALER further employs an adaptive multi-environment RL strategy that dynamically adjusts instance difficulty and curates the active set of environments to track the model's capability frontier and maintain distributional diversity. This co-adaptation prevents reward sparsity, mitigates overfitting to narrow task patterns, and supports sustained improvement throughout training. Extensive experiments show that SCALER consistently outperforms dataset-based RL baselines across diverse reasoning benchmarks and exhibits more stable, long-horizon training dynamics.

SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning

TL;DR

SCALER tackles waning RL rewards during long-horizon reasoning training by introducing a scalable environment synthesis pipeline and adaptive multi-environment RL. It converts real programming problems into verifiable reasoning environments with controllable difficulty and unbounded instances, and couples this with an online difficulty controller and environment curation to keep training on the model's capability frontier while preserving diversity. Empirical results show SCALER outperforms dataset-based RL baselines across five reasoning benchmarks and exhibits more stable, long-horizon training dynamics. This framework provides a scalable platform to study how environment properties influence RL for reasoning and demonstrates the importance of balancing difficulty with diversity during continual learning.

Abstract

Reinforcement learning (RL) offers a principled way to enhance the reasoning capabilities of large language models, yet its effectiveness hinges on training signals that remain informative as models evolve. In practice, RL progress often slows when task difficulty becomes poorly aligned with model capability, or when training is dominated by a narrow set of recurring problem patterns. To jointly address these issues, we propose SCALER (Synthetic sCalable Adaptive Learning Environment for Reasoning), a framework that sustains effective learning signals through adaptive environment design. SCALER introduces a scalable synthesis pipeline that converts real-world programming problems into verifiable reasoning environments with controllable difficulty and unbounded instance generation, enabling RL training beyond finite datasets while preserving strong correctness guarantees. Building on this, SCALER further employs an adaptive multi-environment RL strategy that dynamically adjusts instance difficulty and curates the active set of environments to track the model's capability frontier and maintain distributional diversity. This co-adaptation prevents reward sparsity, mitigates overfitting to narrow task patterns, and supports sustained improvement throughout training. Extensive experiments show that SCALER consistently outperforms dataset-based RL baselines across diverse reasoning benchmarks and exhibits more stable, long-horizon training dynamics.
Paper Structure (23 sections, 3 equations, 5 figures, 1 table)

This paper contains 23 sections, 3 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data-centric RL.
  4. Difficulty-aware RL.
  5. SCALER
  6. Multi-environment Training Framework
  7. Difficulty Controller
  8. Environment Curation Mechanism
  9. From Code Generation to Reasoning Environments
  10. Extracting Meta Information
  11. Generating and Verifying Testcases
  12. Breadth check.
  13. Deep check.
  14. Calibrating Heuristic Difficulty Levels
  15. Experiments
  16. ...and 8 more sections

Figures (5)

  • Figure 1: An illustrative training example of SCALER. The model interacts with a set of active environments: instances are used for training, online accuracy updates the in-environment difficulty controller, and environments whose learning signal saturates are retired and replaced by new environments via the curation mechanism.
  • Figure 2: Left: average performance across the five evaluation benchmarks during Qwen3-4B-base training, comparing dataset-based baselines (MATH, DeepMath) and SCALER. Right: effective sampling statistics under SCALER, indicating that most sampled instances remain near the model's capability boundary.
  • Figure 3: Accuracy improvements for both Qwen3-4B-base and Qwen3-1.7B-base as environment size increases. Both models show a consistent increase in performance with larger environment sizes.
  • Figure 4: Training dynamics of Qwen3-4B-Base with different numbers of environments. The plot shows that even with smaller environments, the model continues to learn with increasing difficulty levels.
  • Figure 5: Ablation study of SCALER on Qwen3-4B-Base. Validation accuracy over training steps for the full system and two variants that remove difficulty controller or the environment curation mechanism. Both components contribute to stronger and more sustained performance improvements.