RL Fine-Tuning Heals OOD Forgetting in SFT
Hangzhan Jin, Sitao Luan, Sicheng Lyu, Guillaume Rabusseau, Reihaneh Rabbany, Doina Precup, Mohammad Hamdaqa
TL;DR
This work challenges the common belief that RL fine-tuning simply generalizes beyond SFT by showing that OOD reasoning accuracy peaks early during SFT and declines thereafter, with RL primarily restoring lost OOD ability within a finite checkpoint window. Through extensive experiments on LLaMA-3.2-11B and Qwen-2.5-8B across multiple reasoning tasks, the authors demonstrate that singular values remain largely stable during fine-tuning, while rotations of singular vectors track OOD forgetting and recovery, revealing a rotation-based mechanism rather than magnitude shifts. They use SVD-based ablations and principal-angle analyses to pinpoint the top-layer and top-k vector directions as critical for OOD behavior, and they show that RL's effectiveness correlates with a relatively flat advantage distribution and the presence of verifiable rewards in non-unique-solution settings. The study thus reframes the two-stage fine-tuning paradigm: RL acts as an automatic mitigation of OOD forgetting introduced by SFT, rather than creating fundamentally new OOD capabilities, and points to rotation-aware fine-tuning as a practical strategy to preserve OOD generalization. The results offer actionable insights for deploying two-stage fine-tuning in practice and provide a principled diagnostic framework for understanding internal model changes during post-training.
Abstract
The two-stage fine-tuning paradigm of Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has empirically shown better reasoning performance than one-stage SFT for the post-training of Large Language Models (LLMs). However, the evolution and mechanism behind the synergy of SFT and RL are still under-explored and inconclusive. In our study, we find the well-known claim "SFT memorizes, RL generalizes" is over-simplified, and discover that: (1) OOD performance peaks at the early stage of SFT and then declines (OOD forgetting), the best SFT checkpoint cannot be captured by training/test loss; (2) the subsequent RL stage does not generate fundamentally better OOD capability, instead it plays an \textbf{OOD restoration} role, recovering the lost reasoning ability during SFT; (3) The recovery ability has boundaries, \ie{} \textbf{if SFT trains for too short or too long, RL cannot recover the lost OOD ability;} (4) To uncover the underlying mechanisms behind the forgetting and restoration process, we employ SVD analysis on parameter matrices, manually edit them, and observe their impacts on model performance. Unlike the common belief that the shift of model capacity mainly results from the changes of singular values, we find that they are actually quite stable throughout fine-tuning. Instead, the OOD behavior strongly correlates with the \textbf{rotation of singular vectors}. Our findings re-identify the roles of SFT and RL in the two-stage fine-tuning and discover the rotation of singular vectors as the key mechanism. %reversing the rotations induced by SFT, which shows recovery from forgetting, whereas imposing the SFT parameter directions onto a RL-tuned model results in performance degradation. Code is available at https://github.com/xiaodanguoguo/RL_Heals_SFT