Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority

TL;DR

The paper tackles the granularity mismatch in Supervised Fine-Tuning (SFT), arguing that aligning data distributions with true human utility requires token-level prioritization rather than uniform likelihood optimization, formalized through a Token Priority function $\Phi$. It presents a density-reshaping view where $\mathcal{P}_{ideal}(x) \propto \Phi(x)\mathcal{P}_{data}(x)$ and recasts training as a priority-weighted loss, enabling two regimes: Positive Priority (construction) and Signed Priority (correction), with Soft Reweighting and Hard Selection as meta-strategies. The framework unifies prior work on filtering, reweighting, and unlearning into a coherent spectrum and highlights practical pathways such as importance sampling for off-policy stability and topology-aware scheduling to improve data efficiency, robustness, and reduction of hallucinations. By reframing SFT as an off-policy, density-shaping problem, the paper argues for dynamic, scalable priority schedules over static data scaling, with meaningful implications for continual learning and RL-like generalization in language-model alignment.

Abstract

The transition from fitting empirical data to achieving true human utility is fundamentally constrained by a granularity mismatch, where fine-grained autoregressive generation is often supervised by coarse or uniform signals. This position paper advocates Token Priority as the essential bridge, formalizing Supervised Fine-Tuning (SFT) not as simple optimization but as a precise distribution reshaping process that aligns raw data with the ideal alignment manifold. We analyze recent breakthroughs through this unified lens, categorizing them into two distinct regimes: Positive Priority for noise filtration and Signed Priority for toxic modes unlearning. We revisit existing progress and limitations, identify key challenges, and suggest directions for future research.