Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training

Xueyan Niu, Bo Bai, Wei Han, Weixi Zhang

TL;DR

The paper investigates whether supervised fine-tuning (SFT) and reinforcement learning (RL) can be decoupled in post-training of large language models. It proves two non-decoupling theorems for the canonical pipelines SFT-then-RL and RL-then-SFT, showing that improvements in one objective (RL reward) inevitably degrade the other (SFT loss), and vice versa. Using information-theoretic tools and a Gibbs-style RL update, the authors establish that any nontrivial RL gain comes at a cost to SFT fitting, and that subsequent SFT after RL reduces RL performance under plausible curvature and shift bounds. Empirical validation on Qwen3-0.6B with CoLA confirms the theoretical predictions: RL increases cross-entropy after SFT, and SFT after RL lowers the RL reward, demonstrating persistent coupling and advocating joint optimization over sequential blocks for post-training strategies.

Abstract

Post-training of large language models routinely interleaves supervised fine-tuning (SFT) with reinforcement learning (RL). These two methods have different objectives: SFT minimizes the cross-entropy loss between model outputs and expert responses, while RL maximizes reward signals derived from human preferences or rule-based verifiers. Modern reasoning models have widely adopted the practice of alternating SFT and RL training. However, there is no theoretical account of whether they can be decoupled. We prove that decoupling is impossible in either order: (1) SFT-then-RL coupling: RL increases SFT loss under SFT optimality and (2) RL-then-SFT coupling: SFT lowers the reward achieved by RL. Experiments on Qwen3-0.6B confirm the predicted degradation, verifying that SFT and RL cannot be separated without loss of prior performance in the post-training

On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training

TL;DR

The paper investigates whether supervised fine-tuning (SFT) and reinforcement learning (RL) can be decoupled in post-training of large language models. It proves two non-decoupling theorems for the canonical pipelines SFT-then-RL and RL-then-SFT, showing that improvements in one objective (RL reward) inevitably degrade the other (SFT loss), and vice versa. Using information-theoretic tools and a Gibbs-style RL update, the authors establish that any nontrivial RL gain comes at a cost to SFT fitting, and that subsequent SFT after RL reduces RL performance under plausible curvature and shift bounds. Empirical validation on Qwen3-0.6B with CoLA confirms the theoretical predictions: RL increases cross-entropy after SFT, and SFT after RL lowers the RL reward, demonstrating persistent coupling and advocating joint optimization over sequential blocks for post-training strategies.

Abstract

Post-training of large language models routinely interleaves supervised fine-tuning (SFT) with reinforcement learning (RL). These two methods have different objectives: SFT minimizes the cross-entropy loss between model outputs and expert responses, while RL maximizes reward signals derived from human preferences or rule-based verifiers. Modern reasoning models have widely adopted the practice of alternating SFT and RL training. However, there is no theoretical account of whether they can be decoupled. We prove that decoupling is impossible in either order: (1) SFT-then-RL coupling: RL increases SFT loss under SFT optimality and (2) RL-then-SFT coupling: SFT lowers the reward achieved by RL. Experiments on Qwen3-0.6B confirm the predicted degradation, verifying that SFT and RL cannot be separated without loss of prior performance in the post-training
Paper Structure (15 sections, 4 theorems, 35 equations, 3 figures)

This paper contains 15 sections, 4 theorems, 35 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Notation
  3. Preliminary
  4. Next-token-prediction Loss
  5. TV and KL Inequalities
  6. Problem Setup
  7. Supervised Fine-Tuning
  8. Reinforcement Learning
  9. Assumptions
  10. SFT-then-RL Coupling
  11. RL-then-SFT Coupling
  12. Empirical Results
  13. SFT-then-RL
  14. RL-then-SFT
  15. Conclusion

Key Result

Lemma 1

The autoregressive training loss eq:sft-loss has the equivalent expression

Figures (3)

  • Figure 1: Training pipeline for modern LLMs. This work focuses on two post-training methods, Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), that refine a pretrained base model after its initial pretraining phase.
  • Figure 2: Any combination of SFT and RL in post-training reduces to the two canonical pipelines: (a) SFT-then-RL and (b) RL-then-SFT.
  • Figure 3: Experimental evidence of coupling. (a) SFT-then-RL: SFT loss climbs immediately once GRPO starts and eventually exceeds the base-model baseline. (b) RL-then-SFT: reward collapses as soon as SFT begins and falls below the base-model level eventually.

Theorems & Definitions (4)

  • Lemma 1
  • Theorem 3.1
  • Proposition 1
  • Theorem 4.1