Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets

TL;DR

This work proposes XTF, an explainable token-level noise filtering framework that decomposes the complex and subtle contributions of token-level data to the fine-tuning process into three distinct and explicit attributes, which can be assessed using scoring methods, and then masks the gradients of selected noisy tokens accordingly to optimize the performance of fine-tuned LLMs.

Abstract

Large Language Models (LLMs) have seen remarkable advancements, achieving state-of-the-art results in diverse applications. Fine-tuning, an important step for adapting LLMs to specific downstream tasks, typically involves further training on corresponding datasets. However, a fundamental discrepancy exists between current fine-tuning datasets and the token-level optimization mechanism of LLMs: most datasets are designed at the sentence-level, which introduces token-level noise, causing negative influence to final performance. In this paper, we propose XTF, an explainable token-level noise filtering framework. XTF decomposes the complex and subtle contributions of token-level data to the fine-tuning process into three distinct and explicit attributes (reasoning importance, knowledge novelty, and task relevance), which can be assessed using scoring methods, and then masks the gradients of selected noisy tokens accordingly to optimize the performance of fine-tuned LLMs. We conduct extensive experiments on three representative downstream tasks (math, code and medicine) across 7 mainstream LLMs. The results demonstrate that XTF can significantly improve downstream performance by up to 13.7% compared to regular fine-tuning. Our work highlights the importance of token-level dataset optimization, and demonstrates the potential of strategies based on attribute decomposition for explaining complex training mechanisms.

Figures (12)

• Figure 1: The accuracy performance of our method on different LLMs. The results show that our method can significantly improve the final performance of fine-tuned LLMs across almost every case.
• Figure 2: The pipeline of token-level data filtering comprises three steps. In the first step, we preprocess the dataset based on a regular format function. In the second phase, we get the sentence-level data item and assess three scores, i.e., attention score, PCP score and relevance score, for the tokens of output label, suggesting the selection of noisy tokens. In the third phase, we mask the noisy tokens and fine-tune the target LLM.
• Figure 3: Distribution of the three scores across different datasets on Deepseek-1.5B. The reasoning importance score is distributed across some fixed values, the knowledge novelty score has a somewhat uniform distribution, and the task relevance score's distribution exhibits clustering features. More distribution figures of different LLMs are shown Appendix \ref{['sec:distribution figures']}.
• Figure 4: Complementarity among attributes using Deepseek-1.5B and GSM8k. Using RI as an example, Only RI represents the percentage of tokens that can be filtered using only RI. After KN represents the percentage of tokens that RI can filter after KN is applied. The reduction in tokens when comparing After KN with Only RI indicates the tokens on which KN and RI overlap.
• Figure 5: The results on code task. We show the results of pass@1, pass@5 and pass@10 respectively.

Theorems & Definitions (23)

• Remark 1: Proof of Assumption \ref{['assump:A5']}
• Lemma 1: Filtering–induced mixture and gradient identities
• proof
• Lemma 2: Explicit alignments under unfiltered and filtered sampling
• proof
• Theorem 1: Filtering strictly improves $M$–alignment (strong MAR case)
• proof
• Theorem 2: Robust alignment gain under bounded selection bias
• proof
• Remark 2: Edge cases