Online SFT for LLM Reasoning: Surprising Effectiveness of Self-Tuning without Rewards

TL;DR

This work introduces Online SFT (OSFT), a reward-free, online self-generated data finetuning method to improve LLM reasoning. OSFT alternates self-data generation at a low sampling temperature with supervised fine-tuning at a separate training temperature, using a loss that reinforces the model’s pretraining-based preferences rather than learning new rewards. Experiments show OSFT achieves downstream performance comparable to strong RLVR baselines like GRPO across six mathematical benchmarks, with notable data efficiency and robustness demonstrated via extensive ablations. The approach offers a simple, practical alternative to reward-based methods, maintaining effectiveness across math-specific and general base models while highlighting the importance of decoupled temperature dynamics in stabilizing learning.

Abstract

We present a simple, self-help online supervised finetuning (OSFT) paradigm for LLM reasoning. In this paradigm, the model generates its own responses and is immediately finetuned on this self-generated data. OSFT is a highly efficient training strategy for LLM reasoning, as it is reward-free and uses just one rollout by default. Experiment results show that OSFT achieves downstream performance on challenging mathematical reasoning tasks comparable to strong reinforcement learning with verifiable rewards (RLVR) methods such as GRPO. Our ablation study further demonstrates the efficiency and robustness of OSFT. The major mechanism of OSFT lies in facilitating the model's own existing preference (latent knowledge) learned from pretraining, which leads to reasoning ability improvement. We believe that OSFT offers an efficient and promising alternative to more complex, reward-based training paradigms. Our code is available at https://github.com/ElementQi/OnlineSFT.

Figures (15)

• Figure 1: Motivation and performance of Online SFT (OSFT) with Qwen2.5-Math-7B as the base model. (a) The performance of the untuned base model is sensitive to different evaluation temperatures ($\tau_{eval}$). Pass@1 accuracy (left) peaks at low temperatures, while pass@8 (right) benefits from moderate temperatures. This motivates a training approach that reinforces the model's existing preference learned from pretraining, which leads to OSFT. (b) OSFT exhibits downstream performance comparable to the baseline GRPO across six math reasoning benchmarks, and both improve over the base model's best pass@1 performance. This reveals a self-weak-to-strong phenomenon, where the real self-help process without any reward in OSFT leads to improvement over the base model. Here, the "Best (Base Model)" horizontal line represents the averaged peak score on these six benchmarks of the untuned base model found by sweeping evaluation temperatures $\tau_{eval}$.
• Figure 2: Given the same question $q$, the base model generates the reasoning steps $[A,B]$ with $B$ being the wrong response (highlighted in light red, picked from one of all 8 wrong tries), while the OSFT model generates the path $[\hat{A},\hat{B}]$ with $\hat{B}$ containing the correct response (highlighted in light blue). It can be seen that OSFT facilitates the base model's existing preference obtained from pretraining, which largely widens the probability margin between the correct path and its counterparts, leading to correct reasoning. The full question and responses are put in \ref{['sec:full_QA']}. More detailed analysis of this experiment result can be found in \ref{['subsubsec:certainty']}.
• Figure 3: PPL of models trained using OSFT and RLVR (GRPO, DAPO, and Dr. GRPO), where the base model is Qwen2.5-Math-7B.
• Figure 4: Performance of OSFT and RLVR (GRPO) on Qwen2.5-Math 1.5B (dashed lines) and 7B (solid lines) models on six math reasoning benchmarks. Dotted horizontal lines represent the best performance of the corresponding base models (before training) achieved by sweeping the evaluation temperature. It can be observed that OSFT is highly competitive with RLVR (GRPO) across different model scales on different math benchmarks.
• Figure 5: Performance of OSFT and RLVR (GRPO) on the Qwen2.5-7B base model across six math reasoning benchmarks. OSFT holds a comparable performance compared to RLVR (GRPO).