Prompt Curriculum Learning for Efficient LLM Post-Training

TL;DR

Prompt Curriculum Learning (PCL) addresses the data-inefficiency of RL-based post-training for large language models by using an online value model to filter prompts with intermediate difficulty, thereby avoiding costly rollouts. The work reveals two actionable insights: there is an optimal batch-size transition point for generation-time scaling, and prompts around $p(x)\approx0.5$ provide the strongest gradient signals. PCL implements an on-policy, rollout-free prompt-filtering mechanism that updates both the policy and the value model, achieving faster convergence and significant compute savings on reasoning benchmarks such as MATH and DeepScaleR. Across these datasets, PCL attains state-of-the-art performance or substantial efficiency gains, demonstrating a practical path to more efficient reasoning-focused RL for LLM post-training while maintaining an on-policy training regime.

Abstract

We introduce Prompt Curriculum Learning (PCL), a lightweight reinforcement learning (RL) algorithm that selects intermediate-difficulty prompts using a learned value model to post-train language models. Since post-training LLMs via RL remains sensitive to batching and prompt selection strategies, we first conduct a series of systematic experiments where we (1) determine the optimal training batch size that balances generation efficiency and gradient quality and (2) establish the importance of focusing on prompts of intermediate difficulty for the policy. We build upon these results to design PCL, which identifies prompts of intermediate difficulty for the current policy in an on-policy manner by using a value model that is concurrently updated based on the current policy. By focusing on informative prompts that yield high effective ratios, PCL achieves either the highest performance or requires significantly less time to reach comparable performance to its counterparts. Compared to rollout-based filtering methods, PCL avoids costly rollouts and achieves $12.1\times$ and $16.9\times$ faster speed on identifying intermediate-difficulty prompts when training on MATH and DeepScaleR, respectively. We further demonstrate that our value model accurately predicts prompt difficulty and allows PCL to focus on progressively more challenging prompts during RL. Our results present a new methodology that delivers improved tradeoff between upper-bound performance and efficiency for reasoning-focused RL.

Figures (18)

• Figure 1: We conduct a systematic investigation of the trade-offs on generation time vs. batch size and number of prompts vs. generations per prompt. We identify an optimal batch size that achieves the best trade-off and discover that the prompts of intermediate difficulty are the most effective for learning. Building on these insights, we introduce Prompt Curriculum Learning (PCL), which trains a value model online for prompt filtering. Compared to the rollout-based filter method, PCL is $12.1\times$ and $16.9\times$ faster during prompt filtering when training on MATH and DeepScaleR respectively.
• Figure 2: (Left / Middle) Training reward as a function of training steps and wall-clock time for Qwen3-4B-Base on MATH and DeepScaleR. The legend indicates the batch configuration in terms of (number of prompts $m$, generations per prompt $n$). (Right) Generation time per step and test accuracy across different batch sizes. The dashed line represents the linear increase that intercepts the origin and the generation time for the largest batch size. Both axes are in log scale. For key takeaways, refer to the paragraph headers in Section \ref{['sec:optimal_bs']}.
• Figure 3: Generation time per step and test accuracy across different batch size combinations (number of prompts $m$, generations per prompt $n$) for Qwen3-4B-Base on MATH and DeepScaleR.
• Figure 4: (Left) Training reward before downsampling in terms of step with number of prompts $m=512$ and generations per prompt $n=8$. (Middle) Training reward after downsampling. (Right) Average effective ratio and gradient norm over training steps, and average test accuracy of six benchmarks across different thresholds. For key takeaways, refer to Section \ref{['sec:optimal_nm']}.
• Figure 5: Average effective ratio over training steps and average test accuracy of six benchmarks under different thresholds $p(x)$ and generations per prompt $n$.