Anchored Supervised Fine-Tuning

TL;DR

This work analyzes the post-training trade-off between supervised fine-tuning (SFT) and reinforcement learning (RL) for large language models, showing that Dynamic Fine-Tuning (DFT) tightens the RL lower bound via a specific auxiliary distribution but suffers from distributional drift and instability. Framing DFT within the Reward-Weighted Regression (RWR) framework reveals its theoretical strengths and the root cause of its instability, motivating Anchored Supervised Fine-Tuning (ASFT) which adds lightweight KL regularization against a base policy to preserve tightness while stabilizing training. Empirically, ASFT consistently outperforms SFT and DFT across mathematical reasoning, medical knowledge grounding, and code generation, with substantial gains at modest data and computation costs, and it scales across model sizes. Additionally, ASFT serves as a strong initialization for subsequent RL fine-tuning, narrowing the gap to RL methods while maintaining SFT-level efficiency, thanks to its principled anchoring and tight lower-bounding properties.

Abstract

Post-training of large language models involves a fundamental trade-off between supervised fine-tuning (SFT), which efficiently mimics demonstrations but tends to memorize, and reinforcement learning (RL), which achieves better generalization at higher computational cost. Dynamic Fine-Tuning (DFT) recently emerged as a promising middle ground, reweighting SFT objectives with token probabilities and achieving improvements in certain reasoning domains, though it exhibits instability in other tasks. We provide a analysis of DFT through the reward-weighted regression (RWR) framework, revealing that it corresponds to a specific auxiliary distribution choice that yields provably tighter RL bounds than standard SFT. However, our analysis also uncovers a critical limitation: this construction lacks distributional anchoring, leading to progressive drift that undermines training stability. To address this, we propose Anchored Supervised Fine-Tuning (ASFT), which augments DFT's reweighting with lightweight KL regularization to preserve tightness while ensuring stability. Empirically, ASFT consistently outperforms both SFT and DFT across mathematical reasoning, medical knowledge grounding, and code generation, achieving substantial improvements with minimal computational overhead. Our RWR framework provides a systematic lens for understanding post-training methods and demonstrates that principled theoretical analysis leads to both stronger guarantees and practical gains.

Figures (9)

• Figure 1: Training dynamics comparison across fine-tuning methods on medical knowledge tasks. Left: MMLU accuracy (out-of-domain evaluation); Center: MedMCQA accuracy (in-domain evaluation); Right: KL divergence from base model. DFT exhibits severe distributional drift (high KL divergence) while ASFT maintains stability through KL anchoring and achieves superior performance on both tasks.
• Figure 2: Comparison of reverse KL (RKL) and forward KL divergence regularization effects across different coefficient values ($\lambda$) on MedQA, MMLU, and MedMCQA benchmarks. Performance is measured in accuracy scores, with horizontal dashed lines indicating baseline performance of Base, SFT, and DFT models for reference.
• Figure 3: Comparison of model performance across three benchmarks (MedQA, MMLU, MedMCQA) for five models (LLaMA-2-7B, LLaMA-2-70B, Qwen2.5-7B, Qwen2.5-32B, Qwen2.5-72B) using four fine-tuning strategies (Base, SFT, DFT, ASFT). Each subplot shows the scores for a specific benchmark, highlighting the relative effectiveness of different fine-tuning methods across models.
• Figure 4: Token probability distribution across different models on the numina_cot training dataset. The x-axis shows the probability intervals of generated tokens, while the y-axis reports the percentage of tokens falling into each interval on a logarithmic scale.
• Figure 5: Ablation study of ASFT on MedCAQA (10k) using LLaMA-2-7B. Right: Effect of batch size, indicating stable performance across a wide range (32–256). ASFT consistently outperforms SFT and DFT under all settings. Left: Effect of different learning rates on average performance, showing that intermediate rates (1e-5 and 1e-4) achieve the best results.

Theorems & Definitions (4)

• Proposition 1: SFT as RL Lower Bound
• Theorem 1: Strict Tightness
• proof : Proof of Proposition \ref{['prop:sft_bound']}
• proof : Proof of Theorem \ref{['thm:tightness']}