RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimization

TL;DR

This work tackles the heavy computational cost of reinforcement fine-tuning for large language models by introducing RPO, a partial reasoning optimization that uses an experience cache of suffixes to guide rollout. By truncating reasoning paths and reusing prefixes from past high-reward trajectories, RPO reduces token generation during rollout and stabilizes gradients, achieving up to ~90% training time reductions while maintaining performance comparable to full-path methods. The approach is plug-and-play, compatible with GRPO and DAPO, and is complemented by length-aware reward shaping to preserve gradient signal diversity. Experiments on 1.5B and 7B models across six reasoning benchmarks demonstrate significant speedups with robust performance, though some reduction in exploration diversity is acknowledged as a trade-off.

Abstract

Within the domain of large language models, reinforcement fine-tuning algorithms necessitate the generation of a complete reasoning trajectory beginning from the input query, which incurs significant computational overhead during the rollout phase of training. To address this issue, we analyze the impact of different segments of the reasoning path on the correctness of the final result and, based on these insights, propose Reinforcement Fine-Tuning with Partial Reasoning Optimization (RPO), a plug-and-play reinforcement fine-tuning algorithm. Unlike traditional reinforcement fine-tuning algorithms that generate full reasoning paths, RPO trains the model by generating suffixes of the reasoning path using experience cache. During the rollout phase of training, RPO reduces token generation in this phase by approximately 95%, greatly lowering the theoretical time overhead. Compared with full-path reinforcement fine-tuning algorithms, RPO reduces the training time of the 1.5B model by 90% and the 7B model by 72%. At the same time, it can be integrated with typical algorithms such as GRPO and DAPO, enabling them to achieve training acceleration while maintaining performance comparable to the original algorithms. Our code is open-sourced at https://github.com/yhz5613813/RPO.

Figures (8)

• Figure 1: The top figure shows the DeepSeekR1-Qwen-Distill-7b and DeepSeekR1-Qwen-Distill-1.5b models. For each question, an initial answer is generated and then truncated; from the truncation point, 256 answers are subsequently generated, and the relationship between truncation length and the overall average accuracy is analyzed. The bottom figure shows 256 answers generated for each training question. Answers exceeding 2048 tokens are selected, and BERT is used to measure the similarity between equal-length prefix segments. The similarity metric is defined as: $\text{sim} = \frac{2}{n(n-1)} \sum_{i=1}^{n-1} \sum_{j=i+1}^{n} \frac{\text{BERT}(s_i) \cdot \text{BERT}(s_j)^\top}{\|\text{BERT}(s_i)\| \, \|\text{BERT}(s_j)\|}$.
• Figure 2: Overview of the RPO framework. The entire training process is described as follows: Cached answer fragments are used by the model to generate new responses; either the best or a random response is selected based on the reward system for optimization; and the cache is continuously updated to improve training efficiency and stability.
• Figure 3: The impact of maximum truncation length $L$ and group size $G$ on the acceleration ratio of POER under the 1.5B and 7B model settings.
• Figure 4: Response length of RPO and GRPO with full-trajectory optimization on 1.5B model across training steps.
• Figure 5: Response length of RPO and GRPO with full-trajectory optimization on 7B model across training steps.