Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning

Ali Taheri, Alireza Taban, Shanshan Ye, Abdolreza Mirzaei, Tongliang Liu, Bo Han

TL;DR

The paper tackles the data quality bottleneck in supervised fine-tuning of large language models by introducing a token-level forgetting mechanism. It scores tokens using cross-model influence to separate them into positive (informative) and negative (noisy) sets, applies standard learning to positives, and forgets negatives through an adaptive regularization term in the training objective. The approach preserves data scale while establishing clearer knowledge boundaries, yielding consistent gains across multiple model sizes (including LLaMA, Qwen, and GPT-Neo) and diverse benchmarks. These results highlight a practical pathway to improve generalization in SFT and point toward a closer integration between forgetting-based regularization and preference-optimization frameworks in LLM fine-tuning.

Abstract

Supervised fine-tuning (SFT) plays a critical role for pretrained large language models (LLMs), notably enhancing their capacity to acquire domain-specific knowledge while preserving or potentially augmenting their general-purpose capabilities. However, the efficacy of SFT hinges on data quality as well as data volume, otherwise it may result in limited performance gains or even degradation relative to the associated baselines. To mitigate such reliance, we suggest categorizing tokens within each corpus into two parts -- positive and negative tokens -- based on whether they are useful to improve model performance. Positive tokens can be trained in common ways, whereas negative tokens, which may lack essential semantics or be misleading, should be explicitly forgotten. Overall, the token categorization facilitate the model to learn less informative message, and the forgetting process shapes a knowledge boundary to guide the model on what information to learn more precisely. We conduct experiments across diverse and well-established benchmarks using various model architectures, demonstrating that this forgetting mechanism enhances model performance.

Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning

TL;DR

The paper tackles the data quality bottleneck in supervised fine-tuning of large language models by introducing a token-level forgetting mechanism. It scores tokens using cross-model influence to separate them into positive (informative) and negative (noisy) sets, applies standard learning to positives, and forgets negatives through an adaptive regularization term in the training objective. The approach preserves data scale while establishing clearer knowledge boundaries, yielding consistent gains across multiple model sizes (including LLaMA, Qwen, and GPT-Neo) and diverse benchmarks. These results highlight a practical pathway to improve generalization in SFT and point toward a closer integration between forgetting-based regularization and preference-optimization frameworks in LLM fine-tuning.

Abstract

Supervised fine-tuning (SFT) plays a critical role for pretrained large language models (LLMs), notably enhancing their capacity to acquire domain-specific knowledge while preserving or potentially augmenting their general-purpose capabilities. However, the efficacy of SFT hinges on data quality as well as data volume, otherwise it may result in limited performance gains or even degradation relative to the associated baselines. To mitigate such reliance, we suggest categorizing tokens within each corpus into two parts -- positive and negative tokens -- based on whether they are useful to improve model performance. Positive tokens can be trained in common ways, whereas negative tokens, which may lack essential semantics or be misleading, should be explicitly forgotten. Overall, the token categorization facilitate the model to learn less informative message, and the forgetting process shapes a knowledge boundary to guide the model on what information to learn more precisely. We conduct experiments across diverse and well-established benchmarks using various model architectures, demonstrating that this forgetting mechanism enhances model performance.

Paper Structure

This paper contains 32 sections, 7 equations, 1 figure, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Data selection for SFT
  4. LLM unlearning and PO
  5. Preliminary
  6. SFT
  7. Data Quality of SFT
  8. Method
  9. Token quality assessment
  10. Token selection
  11. Training objective
  12. Experiments
  13. Experimental setups
  14. Datasets
  15. Training data.
  16. ...and 17 more sections

Figures (1)

  • Figure 1: Performance analysis: (a) Average performance of forgetting versus ignoring methods across different $\rho$ values. (b) Average performance of the forgetting method with different forget rates.