Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

TL;DR

The paper tackles the mismatch between offline SFT and online RL in reasoning LLM post-training. It introduces PEAR, an off-policy evaluation–inspired loss reweighting scheme that uses sequence-, token-, and block-level importance weights to align offline training with RL rollouts, while optionally leveraging negative examples and numerical stabilization. Through extensive experiments on logic games and math reasoning across multiple model families, PEAR consistently improves post-RL performance under fixed RL budgets, sometimes dramatically (e.g., up to 14.6% Pass@8 on AIME-2025), and demonstrates transfer to shifted RL task distributions. The work argues for holistic post-training design, showing that offline gains are not reliable predictors of RL gains and that weighting strategies informed by target policy plausibility yield better RL readiness and practical impact.

Abstract

Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone. We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts. We propose PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected. We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.

Figures (11)

• Figure 1: Offline v.s. Online pass@1 on SynLogic Games on a total of 19 Models. It exhibits significant ranking changes indicating offline performance will not entail online performance. In addtion, our proposed approach remains the most effective initialization for online RL.
• Figure 2: A sketch of our weighing intuition. Red numbers are probabilities under target policy, Blue numbers are probabilities under behavior policy. After token $A$, the behavior (data-generating) policy often continues with $A\!\rightarrow\!B\!\rightarrow\!C$ (e.g., $0.9\times0.8$), but this continuation is highly unlikely for the policy we ultimately want to optimize. As a result, the offline data over-represents $A\!\rightarrow\!B\!\rightarrow\!C$, which can push the model to associate $A$ with an implausible continuation. During online RL, once the model generates $A$, it will rarely follow with $B$ and $C$, so learning from these offline continuations provides little useful signal. We therefore down-weight token $A$ to avoid visiting it.
• Figure 3: An illustration of how PEAR's 3 variant works.
• Figure 4: Results on SynLogic dataset. We demonstrate that PEAR consistently improves post-RL performance.The bars reflect Pass@1 and dots mark Pass@8. In Figure \ref{['fig:synlogic_results_basic']}, Single Step IS is a baseline that corrects each token only based on the probability ratio of the token itself. See \ref{['sec:single_action']}.
• Figure 5: PEAR-to-base KL divergence across weight levels.$y$-axis is the weight (clipped). Token distribution is more heavily driven on important tokens that drives success probability.