Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning

TL;DR

This paper investigates token quality from a noisy-label perspective and proposes a generic token cleaning pipeline for SFT tasks, which filters out uninformative tokens while preserving those carrying key task-specific information.

Abstract

Recent studies show that in supervised fine-tuning (SFT) of large language models (LLMs), data quality matters more than quantity. While most data cleaning methods concentrate on filtering entire samples, the quality of individual tokens within a sample can vary significantly. After pre-training, even in high-quality samples, patterns or phrases that are not task-related can be redundant, uninformative, or even harmful. Continuing to fine-tune on these patterns may offer limited benefit and even degrade downstream task performance. In this paper, we investigate token quality from a noisy-label perspective and propose a generic token cleaning pipeline for SFT tasks. Our method filters out uninformative tokens while preserving those carrying key task-specific information. Specifically, we first evaluate token quality by examining the influence of model updates on each token, then apply a threshold-based separation. The token influence can be measured in a single pass with a fixed reference model or iteratively with self-evolving reference models. The benefits and limitations of both methods are analyzed theoretically by error upper bounds. Extensive experiments show that our framework consistently improves downstream performance. Code is available at https://github.com/UCSC-REAL/TokenCleaning.

Figures (5)

• Figure 1: Overview of the token cleaning pipeline.Fixed-Model Cleaning applies a one-shot cleaning process to the entire dataset $\widetilde{D}$. In contrast, Self-Evolving Cleaning follows an iterative approach. It begins with a warm-up phase, where a model is fine-tuned on the full tokens of split-$0$, denoted as $\widetilde{D}_0$, and then used to clean the next data split, transforming $\widetilde{D}_1$ into $\widehat{D}_1$. The reference model is subsequently updated by fine-tuning the warm-up model (i.e., the first reference model) on $\widehat{D}_1$. This iterative process continues, progressively refining the reference model with each newly cleaned data split.
• Figure 2: Average performance results of two cleaning pipelines under different token proportions. Base model: LLaMA-3.2-3B.
• Figure 3: LLaMA-3.2-3B
• Figure 4: LLaMA-3.1-8B
• Figure 5: Mistral-7B-v0.3

Theorems & Definitions (2)

• Theorem 5.1: Error of learning with full tokens
• Corollary 5.2