ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning

TL;DR

ssToken tackles token-level data quality in LLM supervised fine-tuning by eliminating dependence on external reference models and supplementing loss-based selection with semantic cues from attention. It introduces Retrospective Excess Loss ($REL$) for self-modulated token selection and an attention-based token importance metric ($AttnScore$), combining them with a balance parameter $\gamma$ to select the top-$\rho$ tokens per sample. Across 3B–14B transformers and diverse benchmarks, ssToken surpasses full-data fine-tuning and prior token-selection methods, with reported average gains of up to 4.3% and efficient training. The work provides practical guidance on hyperparameters, demonstrates compatibility with efficient attention mechanisms, and points to future work on adaptive token-budgeting. The contribution is a scalable, data-efficient approach to improve instruction-following and knowledge tasks in LLM fine-tuning.

Abstract

Data quality plays a critical role in enhancing supervised fine-tuning (SFT) for large language models (LLMs), and token-level data selection has emerged as a promising direction for its fine-grained nature. Despite their strong empirical performance, existing token-level selection methods share two key limitations: (1) requiring training or accessing an additional reference model, and (2) relying solely on loss information for token selection, which cannot well preserve semantically important tokens that are not favored by loss-based metrics. To address these challenges, we propose ssToken, a Self-modulated and Semantic-aware Token Selection approach. ssToken leverages readily accessible history models to compute the per-token loss difference with the current model, which serves as a self-modulated signal that enables the model to adaptively select tokens along its optimization trajectory, rather than relying on excess loss from an offline-trained reference model as in prior works. We further introduce a semantic-aware, attention-based token importance estimation metric, orthogonal to loss-based selection and providing complementary semantic information for more effective filtering. Extensive experiments across different model families and scales demonstrate that both self-modulated selection and semantic-aware selection alone outperform full-data fine-tuning, while their integration--ssToken--achieves synergistic gains and further surpasses prior token-level selection methods, delivering performance improvements while maintaining training efficiency.

Figures (3)

• Figure 1: The overall framework of ssToken, which consists of two key components: (i) the self-modulated token selection strategy and (ii) the semantic-aware token importance estimation. We also provide a comparison with prior reference–model–based methods.
• Figure 2: Average performance vs. total training time across different methods.
• Figure 3: Ablation studies on $\gamma$ and $\rho$. (a) Varying the balance coefficient $\gamma$, with dashed lines indicating the full-data fine-tuning baselines for different model sizes. (b) Varying the token selection ratio $\rho$. Both plots report average performance across model sizes (3B, 8B, 14B).