Debunk the Myth of SFT Generalization
Xiaofeng Lin, Hejian Sang, Zhipeng Wang, Xuezhou Zhang
TL;DR
The paper challenges the prevailing belief that supervised fine-tuning (SFT) inherently undergeneralizes compared with reinforcement learning (RL) by showing that data design—especially prompt diversity and chain-of-thought (CoT) supervision—enables vanilla SFT to generalize robustly on decision-making tasks. Through Sokoban and General Points, it demonstrates that a frozen-prompt baseline suffers from surface memorization, which prompt diversification can break, while CoT provides the necessary reasoning scaffolding for harder problems. The key finding is that Diversity + CoT yields the best generalization across instruction-variant and difficulty-variant settings, often matching or surpassing RL baselines while preserving SFT’s simplicity and stability. These results advocate a data-centric approach to post-training, suggesting that with carefully curated demonstrations, SFT can be a practical, competitive alternative to RL in many contexts, though broader validation is needed across modalities and tasks.
Abstract
A prevailing view holds that supervised fine-tuning (SFT) memorizes training data and fails to generalize, whereas reinforcement learning (RL) attains broader robustness. We revisit this claim through a systematic evaluation on two decision-making benchmarks, Sokoban and General Points, and arrive at a different conclusion. We show that much of SFT's perceived failure stems from frozen-prompt artifacts: when trained on fixed instruction templates, SFT models cling to training semantics rather than adapting to new ones. Introducing prompt diversity during training breaks this shortcut and yields strong generalization to unseen instruction variants without harming in-distribution performance. Beyond instruction shifts, we ask whether SFT can generalize to strictly harder tasks. Here, chain-of-thought (CoT) supervision provides an algorithmic scaffold that markedly improves transfer to more difficult regimes, such as larger Sokoban grids with additional boxes and arithmetic with out-of-distribution values or five-card compositions that increase combinatorial complexity. Finally, combining prompt diversity with CoT achieves the best of both worlds: robust generalization across both instruction-variant and difficulty-variant settings, matching or surpassing RL baselines on our benchmarks while retaining SFT's simplicity and stability. These findings challenge the narrative that SFT is inherently inferior to RL and support a data-centric perspective: with appropriately curated demonstrations, vanilla SFT can generalize as strongly as RL. Code reproducing the results in the paper can be found at: https://github.com/XiaofengLin7/debunking-sft-generalization.