Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training

TL;DR

This work directly compares SFT and RFT within continual post-training for multimodal LLMs, showing that RFT inherently mitigates forgetting and preserves general capabilities, approaching multi-task learning performance without data replay. The authors reveal that explicit mechanisms like KL penalties or chain-of-thought reasoning are not the primary drivers of forgetting mitigation; instead, an implicit regularization induced by reward variance curbs updates in regions tied to past knowledge. They provide theoretical intuition via a reward-variance–scaled perspective and introduce RIF-RFT, a rollout-based instance filtering method that improves stability and data efficiency. The findings suggest RFT as a fundamentally more robust paradigm for lifelong adaptation of foundation models, with practical gains demonstrated on seven multimodal tasks and several general benchmarks.

Abstract

Continual post-training (CPT) is a popular and effective technique for adapting foundation models like multimodal large language models to specific and ever-evolving downstream tasks. While existing research has primarily concentrated on methods like data replay, model expansion, or parameter regularization, the fundamental role of the learning paradigm within CPT remains largely unexplored. This paper presents a comparative analysis of two core post-training paradigms: supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT), investigating their respective impacts on knowledge retention during CPT. Our experiments are conducted on a benchmark comprising seven diverse multimodal tasks, utilizing Qwen2.5-VL-7B-Instruct as the base model for continual post-training. The investigation yields two significant findings: (1) When continuously learning on downstream tasks, SFT leads to catastrophic forgetting of previously learned tasks. In contrast, RFT inherently preserves prior knowledge and achieve performance comparable to multi-task training. (2) RFT successfully protects and even enhances the model's general knowledge on standard benchmarks (e.g., MMMU and MMLU-Pro). Conversely, SFT degrades general model capabilities severely. Further analysis reveals that this stability is not primarily due to explicit mechanisms like KL penalty or chain-of-thought reasoning. Instead, we identify an implicit regularization mechanism inherent to RFT as a key contributing factor. Our theoretical analysis suggests that RFT's gradient updates are naturally scaled by the reward variance, acting as a data-dependent regularizer that inherently protects previously acquired knowledge. Finally, we propose a rollout-based instance filtering algorithm to enhance the stability and efficiency of RFT. Our comprehensive study demonstrates the superiority of RFT as a robust paradigm for continual post-training.

Figures (5)

• Figure 1: Comparison of performance retention between SFT and RFT in continual post-training. We plot the performance on each task, normalized relative to its initial post-training peak, as the model learns through a sequence of multimodal tasks. (a) SFT exhibits classic catastrophic forgetting, where performance on previously learned tasks degrades dramatically as new tasks are introduced. (b) By contrast, RFT demonstrates remarkable stability, maintaining high performance on prior tasks throughout the entire sequence. This suggests an inherent forgetting-mitigation property within the RFT paradigm. Further details on the experimental setup can be found in Section \ref{['sec:experiments']}.
• Figure 2: General capability preservation after continual post-training. We evaluate models at the end of learning all downstream tasks on general benchmarks using both CoT and direct prompting. Compared to the base model, SFT (shown in light colors) causes degradation while RFT (shown in darker colors) preserves and even enhances general capabilities.
• Figure 3: Illustrative examples of continual post-training benchmark.
• Figure 4: Loss landscapes illustration of SFT and RFT based continual post-training.
• Figure 5: Example prompt templates w/o and w/ CoT.

Theorems & Definitions (6)

• Definition 5.1: Forgetting Risk
• Proposition 5.2: RFT's Implicit Regularization Effect
• Definition B.1: Importance-Weighted Score Norm (IWSN)
• Lemma B.3: Variance of Advantage
• proof
• proof : Proof of Proposition \ref{['pro:rft_stability']}