Towards a Unified View of Large Language Model Post-Training

TL;DR

This work unifies post-training strategies for large language models by introducing the Unified Policy Gradient Estimator (UPGE), which shows that SFT and RL gradients are instances of a single objective. Building on UPGE, the authors propose Hybrid Post-Training (HPT), a dynamic algorithm that blends SFT and RL signals based on real-time performance feedback. Theoretical derivations reveal the four core gradient components and their bias-variance tradeoffs, while extensive experiments across multiple models and six math benchmarks demonstrate that HPT consistently outperforms strong baselines and sequential approaches. The results highlight the practical value of a principled, adaptive framework that leverages both offline demonstrations and online exploration to improve reasoning and generalization.

Abstract

Two major sources of training data exist for post-training modern language models: online (model-generated rollouts) data, and offline (human or other-model demonstrations) data. These two types of data are typically used by approaches like Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT), respectively. In this paper, we show that these approaches are not in contradiction, but are instances of a single optimization process. We derive a Unified Policy Gradient Estimator, and present the calculations of a wide spectrum of post-training approaches as the gradient of a common objective under different data distribution assumptions and various bias-variance tradeoffs. The gradient estimator is constructed with four interchangeable parts: stabilization mask, reference policy denominator, advantage estimate, and likelihood gradient. Motivated by our theoretical findings, we propose Hybrid Post-Training (HPT), an algorithm that dynamically selects different training signals. HPT is designed to yield both effective exploitation of demonstration and stable exploration without sacrificing learned reasoning patterns. We provide extensive experiments and ablation studies to verify the effectiveness of our unified theoretical framework and HPT. Across six mathematical reasoning benchmarks and two out-of-distribution suites, HPT consistently surpasses strong baselines across models of varying scales and families.

Figures (8)

• Figure 1: Illustration of the Unified Policy Gradient Estimator. The "$\nabla$" in the background of the Likelihood Gradient part refers to the calculation of the gradient with respect to the $\pi_\theta$.
• Figure 2: Pass@k performance of HPT against baselines on Qwen2.5-Math-7B. The evaluation spans 3 benchmarks, with Pass@k values estimated via bootstrap sampling from a set of $2048$ generated solutions per problem.
• Figure 3: GRPO training dynamics of SFT$\rightarrow$GRPO on Qwen2.5-Math-1.5B across 50 training epochs. We visualize the model's per-question sampling accuracy throughout the training process.
• Figure 4: Performance difference (HPT v.s. SFT$\rightarrow$GRPO) on Qwen2.5-Math-1.5B across 50 training epochs. A diverging color scale indicates the advantage: red for HPT, blue for SFT$\rightarrow$GRPO, and white for no difference.
• Figure 5: Validation performance comparisons on Qwen2.5-Math-1.5B across benchmarks.

Theorems & Definitions (4)

• Lemma A1: Score-function identity
• Lemma A2: Differentiating an expectation with parameterized integrand
• Lemma A3: Measure-change (importance reweighting) identity
• proof