Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TACLer: Tailored Curriculum Reinforcement Learning for Efficient Reasoning

Huiyuan Lai, Malvina Nissim

TL;DR

LLMs delivering long chain-of-thought reasoning achieve strong performance but incur high computational cost and overthinking. TACLer introduces a model-tailored curriculum reinforcement learning framework coupled with a hybrid Thinking/NoThinking reasoning paradigm to learn efficiently and reason effectively. It combines tailored curriculum learning, a flexible dual-mode reasoning strategy, and GRPO-based training, validated across four math benchmarks, showing reduced training compute and token usage alongside accuracy gains. This approach offers a scalable, flexible path toward efficient multi-step reasoning with potential applicability beyond mathematics, though it notes inference overhead as a consideration for future work.

Abstract

Large Language Models (LLMs) have shown remarkable performance on complex reasoning tasks, especially when equipped with long chain-of-thought (CoT) reasoning. However, eliciting long CoT typically requires large-scale reinforcement learning (RL) training, while often leading to overthinking with redundant intermediate steps. To improve learning and reasoning efficiency, while preserving or even enhancing performance, we propose TACLer, a model-tailored curriculum reinforcement learning framework that gradually increases the complexity of the data based on the model's proficiency in multi-stage RL training. TACLer features two core components: (i) tailored curriculum learning that determines what knowledge the model lacks and needs to learn in progressive stages; (ii) a hybrid Thinking/NoThinking reasoning paradigm that balances accuracy and efficiency by enabling or disabling the Thinking mode. Our experiments show that TACLer yields a twofold advantage in learning and reasoning: (i) it reduces computational cost, cutting training compute by over 50% compared to long thinking models and reducing inference token usage by over 42% relative to the base model; and (ii) it improves accuracy by over 9% on the base model, consistently outperforming state-of-the-art Nothinking and Thinking baselines across four math datasets with complex problems.

TACLer: Tailored Curriculum Reinforcement Learning for Efficient Reasoning

TL;DR

LLMs delivering long chain-of-thought reasoning achieve strong performance but incur high computational cost and overthinking. TACLer introduces a model-tailored curriculum reinforcement learning framework coupled with a hybrid Thinking/NoThinking reasoning paradigm to learn efficiently and reason effectively. It combines tailored curriculum learning, a flexible dual-mode reasoning strategy, and GRPO-based training, validated across four math benchmarks, showing reduced training compute and token usage alongside accuracy gains. This approach offers a scalable, flexible path toward efficient multi-step reasoning with potential applicability beyond mathematics, though it notes inference overhead as a consideration for future work.

Abstract

Large Language Models (LLMs) have shown remarkable performance on complex reasoning tasks, especially when equipped with long chain-of-thought (CoT) reasoning. However, eliciting long CoT typically requires large-scale reinforcement learning (RL) training, while often leading to overthinking with redundant intermediate steps. To improve learning and reasoning efficiency, while preserving or even enhancing performance, we propose TACLer, a model-tailored curriculum reinforcement learning framework that gradually increases the complexity of the data based on the model's proficiency in multi-stage RL training. TACLer features two core components: (i) tailored curriculum learning that determines what knowledge the model lacks and needs to learn in progressive stages; (ii) a hybrid Thinking/NoThinking reasoning paradigm that balances accuracy and efficiency by enabling or disabling the Thinking mode. Our experiments show that TACLer yields a twofold advantage in learning and reasoning: (i) it reduces computational cost, cutting training compute by over 50% compared to long thinking models and reducing inference token usage by over 42% relative to the base model; and (ii) it improves accuracy by over 9% on the base model, consistently outperforming state-of-the-art Nothinking and Thinking baselines across four math datasets with complex problems.
Paper Structure (31 sections, 2 equations, 9 figures, 7 tables)

This paper contains 31 sections, 2 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminaries: Learning and Reasoning Efficiency
  3. Methodology
  4. Tailored Curriculum Learning
  5. Hybrid Reasoning Mode
  6. Thinking
  7. NoThinking
  8. Training with GRPO
  9. Experiments
  10. Experimental Setup
  11. Model and Dataset
  12. Baselines
  13. Main Results
  14. Reasoning Accuracy and Efficiency in Thinking Mode
  15. Reasoning Accuracy and Efficiency in Saving Mode
  16. ...and 16 more sections

Figures (9)

  • Figure 1: Response clipping ratio in the first training stage of DeepScaleR-1.5B-Preview.
  • Figure 2: Comparison of DeepSeek-R1-Distill-Qwen-1.5B and DeepScaleR-1.5B-Preview using NoThinking mode across different difficulty levels of MATH500 dataset.
  • Figure 3: Overview of TACLer, our tailored curriculum reinforcement learning framework.
  • Figure 4: Comparison of different training stages on different difficulty levels of MATH500. Note that state 0 represents the base model.
  • Figure 5: Comparison of TACLer, Direct-Train, and DeepScaleR-1.5B-Preview in terms of response clip ratio, length, and reward during the first training stage.
  • ...and 4 more figures