Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling

TL;DR

This paper argues that Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) are complementary rather than competing paradigms for post-training LLMs. It introduces Prefix-RFT, a simple hybrid method that samples and prefixes demonstrations to guide on-policy exploration, combining SFT's stable supervision with RFT's task-focused optimization. Through math-reasoning benchmarks, Prefix-RFT consistently outperforms standalone SFT/RFT and other hybrids, with robust performance across model scales and demonstration quality. The work also provides detailed analysis of the dynamic transition between SFT and RFT during training and proposes practical components—entropy-based clipping and a cosine-decay prefix scheduler—to stabilize and improve results. Overall, Prefix-RFT offers a principled, easily integrable direction for unified post-training of reasoning-enabled LLMs with practical benefits for real-world deployment.

Abstract

Existing post-training techniques for large language models are broadly categorized into Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT). Each paradigm presents a distinct trade-off: SFT excels at mimicking demonstration data but can lead to problematic generalization as a form of behavior cloning. Conversely, RFT can significantly enhance a model's performance but is prone to learn unexpected behaviors, and its performance is highly sensitive to the initial policy. In this paper, we propose a unified view of these methods and introduce Prefix-RFT, a hybrid approach that synergizes learning from both demonstration and exploration. Using mathematical reasoning problems as a testbed, we empirically demonstrate that Prefix-RFT is both simple and effective. It not only surpasses the performance of standalone SFT and RFT but also outperforms parallel mixed-policy RFT methods. A key advantage is its seamless integration into existing open-source frameworks, requiring only minimal modifications to the standard RFT pipeline. Our analysis highlights the complementary nature of SFT and RFT, and validates that Prefix-RFT effectively harmonizes these two learning paradigms. Furthermore, ablation studies confirm the method's robustness to variations in the quality and quantity of demonstration data. We hope this work offers a new perspective on LLM post-training, suggesting that a unified paradigm that judiciously integrates demonstration and exploration could be a promising direction for future research.

Figures (7)

• Figure 1: The training pipeline of Prefix-RFT . The method does minimal modification to the existing RFT training pipeline. Given a problem and a demonstration, a prefix is sampled to guide the online continuation. The concatenated sequence $y_n$ is mixed with other online rollouts to perform RFT-style training. We also utilize an entropy-based clipping strategy to constrain the update on demonstration.
• Figure 2: Averaged performance of math reasoning benchmarks of Prefix-RFT and other baselines.
• Figure 3: Training trajectories of SFT, RFT, and Prefix-RFT. The x-axis denotes the SFT loss on the demonstrations. The y-axis represents the Avg@16 and Best@16 scores. The final step is marked with the yellow star, annotated using the final SFT loss and the final score.
• Figure 4: The average reward of rollouts initiated with a prefix and the overall reward. The shaded area represents the advantage assigned to the prefix. As the advantage diminishes, the training gradually transitions from SFT to RFT, aligning with the widely used SFT-then-RFT pipeline
• Figure 5: The change in SFT loss on demonstrations on problems of varying difficulties, suggesting Prefix-RFT provides more supervision for more challenging problems.