Graceful Forgetting in Generative Language Models

TL;DR

The paper tackles negative transfer in the pretrain-finetune paradigm for generative language models by proposing Learning With Forgetting (LWF), a framework that enables graceful forgetting. LWF elicits self-knowledge via self-generated data, evaluates forgetting confidence for each data point with a Fisher Information-based weighting $F_L$, and periodically unlearns high-confidence samples to improve fine-tuning plasticity. The approach is demonstrated on domain-specific QA tasks using Llama-based models, with ablations showing the importance of data-quality via forgetting confidence and the benefits of interleaved learning/unlearning scheduling; results scale to larger models albeit with reduced gains. This work provides a practical mechanism to mitigate detrimental pre-trained knowledge during fine-tuning and offers a foundation for future forgetting-aware optimization in autoregressive language models.

Abstract

Recently, the pretrain-finetune paradigm has become a cornerstone in various deep learning areas. While in general the pre-trained model would promote both effectiveness and efficiency of downstream tasks fine-tuning, studies have shown that not all knowledge acquired during pre-training is beneficial. Some of the knowledge may actually bring detrimental effects to the fine-tuning tasks, which is also known as negative transfer. To address this problem, graceful forgetting has emerged as a promising approach. The core principle of graceful forgetting is to enhance the learning plasticity of the target task by selectively discarding irrelevant knowledge. However, this approach remains underexplored in the context of generative language models, and it is often challenging to migrate existing forgetting algorithms to these models due to architecture incompatibility. To bridge this gap, in this paper we propose a novel framework, Learning With Forgetting (LWF), to achieve graceful forgetting in generative language models. With Fisher Information Matrix weighting the intended parameter updates, LWF computes forgetting confidence to evaluate self-generated knowledge regarding the forgetting task, and consequently, knowledge with high confidence is periodically unlearned during fine-tuning. Our experiments demonstrate that, although thoroughly uncovering the mechanisms of knowledge interaction remains challenging in pre-trained language models, applying graceful forgetting can contribute to enhanced fine-tuning performance.

Figures (6)

• Figure 1: The overview of the LWF framework. Given the forgetting task $\mathcal{D}_F$ and learning task $\mathcal{D}_L$, LWF first constructs $\mathcal{D}_{self}$ through self-generated texts to represent the knowledge regarding the forgetting task. Then, with the Fisher Information Matrix $F_L$ and the optimal parameters of the learning task approximated from $\mathcal{D}_L$, LWF calculates forgetting confidence for each data point in $\mathcal{D}_{self}$. Finally, data points with high forgetting confidence are selected for unlearning, represented by $\mathcal{D}_U$. The unlearning process is integrated into the fine-tuning process of $\mathcal{D}_L$ and is executed periodically at intervals of $N_u$.
• Figure 2: Distribution of accuracy changes between two filtering strategies. The $y$-axis represents distribution density. All percentages are calculated based on vanilla fine-tuning.
• Figure 3: Accuracy change percentage of the forgetting task across different learning-forgetting combinations. Percentages are computed based on vanilla fine-tuning.
• Figure 4: Cosine similarity between the outputs of forgetting tasks generated by the vanilla fine-tuned model and LWF resulting model. Values are multiplied by 100.
• Figure 5: TTR change percentage of the forgetting task across different learning-forgetting combinations. Percentages are computed based on vanilla fine-tuning.