Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math

Bo Pang, Deqian Kong, Silvio Savarese, Caiming Xiong, Yingbo Zhou

TL;DR

Reasoning Curriculum tackles general-purpose reasoning by first eliciting robust math-based reasoning via a brief cold-start and math-only RL, then transferring and refining these skills across diverse domains through joint RL. The approach remains backbone-agnostic and relies on verifiable rewards, using a GRPO/DAPO-based objective with domain-specific verification strategies. Empirical results on Qwen-3-4B and Llama-3.1-8B show consistent multi-domain gains across math, code, STEM, logic, simulation, and tabular tasks, with ablations confirming the necessity of both stages and a cognitive-skill analysis highlighting increased verification and backtracking behaviors. The method offers a pragmatic, easy-to-adopt recipe for building broadly capable reasoning models with practical implications for cross-domain AI systems.

Abstract

Reinforcement learning (RL) can elicit strong reasoning in large language models (LLMs), yet most open efforts focus on math and code. We propose Reasoning Curriculum, a simple two-stage curriculum that first elicits reasoning skills in pretraining-aligned domains such as math, then adapts and refines these skills across other domains via joint RL. Stage 1 performs a brief cold start and then math-only RL with verifiable rewards to develop reasoning skills. Stage 2 runs joint RL on mixed-domain data to transfer and consolidate these skills. The curriculum is minimal and backbone-agnostic, requiring no specialized reward models beyond standard verifiability checks. Evaluated on Qwen3-4B and Llama-3.1-8B over a multi-domain suite, reasoning curriculum yields consistent gains. Ablations and a cognitive-skill analysis indicate that both stages are necessary and that math-first elicitation increases cognitive behaviors important for solving complex problems. Reasoning Curriculum provides a compact, easy-to-adopt recipe for general reasoning.

Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math

TL;DR

Reasoning Curriculum tackles general-purpose reasoning by first eliciting robust math-based reasoning via a brief cold-start and math-only RL, then transferring and refining these skills across diverse domains through joint RL. The approach remains backbone-agnostic and relies on verifiable rewards, using a GRPO/DAPO-based objective with domain-specific verification strategies. Empirical results on Qwen-3-4B and Llama-3.1-8B show consistent multi-domain gains across math, code, STEM, logic, simulation, and tabular tasks, with ablations confirming the necessity of both stages and a cognitive-skill analysis highlighting increased verification and backtracking behaviors. The method offers a pragmatic, easy-to-adopt recipe for building broadly capable reasoning models with practical implications for cross-domain AI systems.

Abstract

Reinforcement learning (RL) can elicit strong reasoning in large language models (LLMs), yet most open efforts focus on math and code. We propose Reasoning Curriculum, a simple two-stage curriculum that first elicits reasoning skills in pretraining-aligned domains such as math, then adapts and refines these skills across other domains via joint RL. Stage 1 performs a brief cold start and then math-only RL with verifiable rewards to develop reasoning skills. Stage 2 runs joint RL on mixed-domain data to transfer and consolidate these skills. The curriculum is minimal and backbone-agnostic, requiring no specialized reward models beyond standard verifiability checks. Evaluated on Qwen3-4B and Llama-3.1-8B over a multi-domain suite, reasoning curriculum yields consistent gains. Ablations and a cognitive-skill analysis indicate that both stages are necessary and that math-first elicitation increases cognitive behaviors important for solving complex problems. Reasoning Curriculum provides a compact, easy-to-adopt recipe for general reasoning.

Paper Structure

This paper contains 23 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Reasoning Curriculum
  3. Math Training
  4. Cold Start
  5. Math RL
  6. Joint RL
  7. Experiments
  8. Training Data
  9. Cold Start Data
  10. Reinforcement Learning Data
  11. Training Setup
  12. Evaluation Benchmarks
  13. Baselines
  14. Results
  15. Results on Qwen
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Reasoning curriculum overview. Stage 0 (pretraining, not conducted in this work): cognitive skills exist but are weakly expressed on data-rich domains like math. Stage 1 (cold-start + math-only RL): skills are elicited and strengthened in pretraining-primed domains. Stage 2 (joint RL): skills are transferred and refined across general domains (code, logic, tabular, simulation). Blue arrows indicate the training progression.
  • Figure 2: Cognitive skill frequencies by training setting. RL = direct joint RL; CS+RL = cold-start then joint RL; RC = reasoning curriculum. Top: Qwen3-4B; bottom: Llama-3.1-8B.
  • Figure 3: Trends across curriculum stages by task. CS = Cold-Start; RL = Math-RL; Joint-RL = RL on mixed-domain data. Top: Qwen3-4B; bottom: Llama-3.1-8B. Each point shows the average score within a domain at each stage.