Efficient Reinforcement Finetuning via Adaptive Curriculum Learning

TL;DR

AdaRFT tackles the inefficiency of reinforcement finetuning for mathematical reasoning by introducing an adaptive curriculum that continuously aligns training difficulty with the model’s evolving capability. It dynamically updates a target difficulty based on reward feedback and selects tasks near that target, enabling faster convergence and better final accuracy without modifying the reward function or underlying RL method. Across multiple data distributions and model sizes, AdaRFT achieves substantial training-time savings and performance gains, particularly in imbalanced data regimes. The approach demonstrates broad compatibility with PPO and other RL algorithms, suggesting practical scalability for structured reasoning tasks beyond mathematics.

Abstract

Reinforcement finetuning (RFT) has shown great potential for enhancing the mathematical reasoning capabilities of large language models (LLMs), but it is often sample- and compute-inefficient, requiring extensive training. In this work, we introduce AdaRFT (Adaptive Curriculum Reinforcement Finetuning), a method that significantly improves both the efficiency and final accuracy of RFT through adaptive curriculum learning. AdaRFT dynamically adjusts the difficulty of training problems based on the model's recent reward signals, ensuring that the model consistently trains on tasks that are challenging but solvable. This adaptive sampling strategy accelerates learning by maintaining an optimal difficulty range, avoiding wasted computation on problems that are too easy or too hard. AdaRFT requires only a lightweight extension to standard RFT algorithms like Proximal Policy Optimization (PPO), without modifying the reward function or model architecture. Experiments on competition-level math datasets-including AMC, AIME, and IMO-style problems-demonstrate that AdaRFT significantly improves both training efficiency and reasoning performance. We evaluate AdaRFT across multiple data distributions and model sizes, showing that it reduces training time by up to 2x and improves accuracy by a considerable margin, offering a more scalable and effective RFT framework.

Figures (8)

• Figure 1: Evaluation of difficulty estimation: (a) Stability of difficulty scores under subsampling of model rollouts; (b) Correlation between labeled difficulty levels and average solved percentage.
• Figure 1: Average time per step (in seconds) at step 100 and extra steps required to match AdaRFT's performance at step 60 (for Qwen 2.5 Math 1.5B) or step 40 (for Qwen 2.5 7B), across different setups and methods.
• Figure 2: Difficulty distribution for different training sets: Uniform, Skew-Difficult, and Skew-Easy. Each training set contains 10,000 samples.
• Figure 3: Performance comparison of PPO, PPO with filtered data, and AdaRFT(PPO) across different setups (uniform, skew-easy, skew-difficult) for Qwen 2.5 Math 1.5B and Qwen 2.5 7B models. Accuracy is the average of MATH 500, GSM8K, AIME 24, AMC 23, OlympiadBench, and Minerva Math. For clarity, curves are exponentially smoothed ($\alpha = 0.3$) to reduce noise.
• Figure 4: Performance comparison of Qwen 2.5 7B trained on different data distributions using PPO (Uniform, Easy-Extreme, Hard-Extreme) and AdaRFT instantiated with PPO (Uniform + AdaRFT). For clarity, curves are exponentially smoothed ($\alpha = 0.3$) to reduce noise.