CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models

TL;DR

This work tackles the high training cost of GRPO-based reasoning models caused by sampling many completions per prompt. It introduces Completion Pruning Policy Optimization (CPPO), which prunes completions based on their absolute advantage $|A_i|$ by enforcing a threshold $\gamma$, thereby reducing forward passes and gradient computations. A dynamic completion allocation strategy further improves GPU utilization by continuously feeding devices with high-value completions from new questions. Empirical results on GSM8K and MATH show up to $7.98\times$ speedup with CPPO while preserving or improving accuracy, and CPPO generalizes to other RL algorithms and backbones, illustrating a practical path to scalable, efficient reasoning model training.

Abstract

This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO). GRPO, while effective, incurs high training costs due to the need to sample multiple completions for each question. Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases training time multiplicatively, and not all completions contribute equally to policy training -- their contribution depends on their relative advantage. To address these issues, we propose CPPO, which prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates. Additionally, we introduce a dynamic completion allocation strategy to maximize GPU utilization by incorporating additional questions, further enhancing training efficiency. Experiments show that CPPO achieves up to $7.98\times$ speedup on GSM8K and $3.48\times$ on Math while preserving or even enhancing the accuracy compared to the original GRPO. We release our code at \href{https://github.com/lzhxmu/CPPO}{https://github.com/lzhxmu/CPPO}.

Figures (6)

• Figure 1: Completion number vs. (left) accuracy and (right) training time. Experiments are conducted on GSM8K cobbe2021gsm8k using Qwen2.5-1.5B-Instruct yang2024qwen2.5.
• Figure 2: Overview of Completion Pruning Policy Optimization (CPPO). After obtaining the advantages, only completions with high absolute advantage are retained for the forward of the policy model, reference model, and old policy model. $\mathbb{D}_{R}$ is the probability ratio of the policy to old policy models.
• Figure 3: Illustration of dynamic completion allocation for parallel processing. After pruning completions, completion allocation incorporates important completions from new questions. The $o_m^n$ represents the $m$-th completion of the $n$-th question.
• Figure 4: Evaluation accuracy comparison. Left: Qwen2.5-1.5b-Instruct on GSM8K test subset. Right: Qwen2.5-7b-Instruct on MATH test subset. Here, $k$ denotes the retained completion quantity by CPPO (or generated by GRPO), and $b$ represents questions per training step.
• Figure 5: Reward comparison during training (1 epoch) between GRPO and CPPO on GSM8K dataset.