GIFT: Unlocking Global Optimality in Post-Training via Finite-Temperature Gibbs Initialization

TL;DR

This work identifies a fundamental mismatch between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) in large reasoning models, arising from a $0$-temperature collapse that hampers exploration. It derives a global post-training objective whose optimal initialization is a finite-temperature Gibbs distribution, and introduces Gibbs Initialization with Finite Temperature (GIFT) to preserve base priors while enabling effective RL exploration. Empirical results on DeepMath-103k and multiple backbones show that GIFT consistently outperforms standard SFT and other baselines, with improved mathematical reasoning, robustness to distributional shifts, and better exploration without sacrificing base knowledge. The approach offers a principled pathway to global optimality in post-training, supported by analyses of geometric and distributional consistency and by finite-temperature theory that explains improved RL convergence and generalization.

Abstract

The prevailing post-training paradigm for Large Reasoning Models (LRMs)--Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL)--suffers from an intrinsic optimization mismatch: the rigid supervision inherent in SFT induces distributional collapse, thereby exhausting the exploration space necessary for subsequent RL. In this paper, we reformulate SFT within a unified post-training framework and propose Gibbs Initialization with Finite Temperature (GIFT). We characterize standard SFT as a degenerate zero-temperature limit that suppresses base priors. Conversely, GIFT incorporates supervision as a finite-temperature energy potential, establishing a distributional bridge that ensures objective consistency throughout the post-training pipeline. Our experiments demonstrate that GIFT significantly outperforms standard SFT and other competitive baselines when utilized for RL initialization, providing a mathematically principled pathway toward achieving global optimality in post-training. Our code is available at https://github.com/zzy1127/GIFT.

Figures (6)

• Figure 1: Pass@k performance comparison. We compare the average accuracy of GIFT versus Standard SFT across varying sample counts ($k$) prior to RL training.
• Figure 2: Impact of inverse temperature $\beta$ on RL performance. Average accuracy on math benchmarks peaks at finite $\beta$ for both models, surpassing the standard SFT baseline (dashed line).
• Figure 3: Top-$K$ Token Overlap Analysis. Results for Llama-3.1-8B (Left) and Qwen2.5-7B (Right). GIFT exhibits consistently higher token overlap than the SFT baseline.
• Figure 4: Ablation study of the uniform distribution smoothing term.
• Figure 5: Average pass@1 accuracy across the RL training progress.