Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spurious Forgetting in Continual Learning of Language Models

Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma

TL;DR

The paper identifies spurious forgetting as a misalignment phenomenon in continual learning of language models, where old-task performance deteriorates due to shifts in task alignment rather than loss of underlying knowledge. It analyzes this via a synthetic Biography dataset and a theoretical orthogonal-update framework, revealing that early optimization steps primarily undo prior task alignment in the bottom layers. A Freezing strategy that fixes bottom layers substantially improves performance across multiple continual-learning scenarios, supported by both loss-landscape and principal-component analyses and validated on real-world settings such as safety alignment and instruction-tuning. While data replay can outperform freezing, Freeze provides a practical, data-efficient, and broadly applicable approach to mitigate spurious forgetting, with guidelines on layer freezing depth depending on task alignment similarity.

Abstract

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

Spurious Forgetting in Continual Learning of Language Models

TL;DR

The paper identifies spurious forgetting as a misalignment phenomenon in continual learning of language models, where old-task performance deteriorates due to shifts in task alignment rather than loss of underlying knowledge. It analyzes this via a synthetic Biography dataset and a theoretical orthogonal-update framework, revealing that early optimization steps primarily undo prior task alignment in the bottom layers. A Freezing strategy that fixes bottom layers substantially improves performance across multiple continual-learning scenarios, supported by both loss-landscape and principal-component analyses and validated on real-world settings such as safety alignment and instruction-tuning. While data replay can outperform freezing, Freeze provides a practical, data-efficient, and broadly applicable approach to mitigate spurious forgetting, with guidelines on layer freezing depth depending on task alignment similarity.

Abstract

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.
Paper Structure (78 sections, 7 theorems, 76 equations, 20 figures, 36 tables)

This paper contains 78 sections, 7 theorems, 76 equations, 20 figures, 36 tables.

Table of Contents

  1. Introduction
  2. Motivation: Preliminary Experiments on Spurious Forgetting
  3. A Closer Look at Spurious Forgetting
  4. Controlled Settings under Synthetic Dataset
  5. Spurious Forgetting from Performance Perspective
  6. Loss Landscape Perspective
  7. Model Weight Perspective
  8. Feature Perspective
  9. Summary
  10. Theoretical Analysis
  11. Solution to Spurious Forgetting
  12. Revisiting Existing Techniques for Forgetting
  13. Mitigating Spurious Forgetting by Freezing Bottom Layers
  14. Application on Real-World Scenarios
  15. Related Work
  16. ...and 63 more sections

Key Result

Proposition 4.6

Consider the mapping $\mathbf{Y} = \mathbf{W} \mathbf{X}$, where $\mathbf{W} \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, and $\mathbf{X} \in \mathbb{R}^{d_{\text{in}} \times n}$. Suppose $\mathbf{W}$ is updated as $\tilde{\mathbf{W}} = \mathbf{W} + \Delta \mathbf{W}$, where $\Delta \mathb

Figures (20)

  • Figure 1: We are the first to investigate "spurious forgetting" in continual learning of LLMs.
  • Figure 2: Spurious Forgetting in the controlled setting. (a) The Spurious Forgetting from performance perspective, Task 0 ACC and Task 1 ACC refer to the first-token accuracy while Recovered Task 0 ACC is the exact match accuracy. (b) and (c) illustrated our experiments of continual learning and recovery on Task 0.
  • Figure 3: The loss landscape of test loss of Task 0 (upper), Task 1 (lower) of two methods: (a) SEQ: sequential finetuning; (b) data replay with 20% of old data. The y-axis is the weight update direction of the initial 150 steps and the x-axis is the weight update direction in the subsequent steps.Full results are in Appendix \ref{['sec:appendix_loss_landscape']}.
  • Figure 4: Angles between model weight updates. $\Delta PT$, $\Delta Task0$, and $\Delta Task1$ denote weight updates from pretraining, finetuning Task 0, and finetuning Task 1 stages, respectively. $\Delta Task0_0^{150}$ represents the weight update from the weight at the 150-th step minus the weight at the 0-th step. Figures (a) and (b) compare the angles between weight updates during pretraining and Task 0, and between Task 0 and Task 1, respectively. Full results are provided in Appendix \ref{['sec:appendix_model_weight']}.
  • Figure 5: The shift of features in principal components. Case1: Finetuning Task 0 (step 0 - final); Case2: Finetuning Task 1 (step 100-150); Case3: Finetuning Task 1 (step 200 - final); Case4: Finetuning Task 1 (step 0 - final). Full results are provided in Appendix \ref{['sec:appendix_feature_shift']}.
  • ...and 15 more figures

Theorems & Definitions (24)

  • Definition 4.1: Residual Network Structure
  • Remark 4.2
  • Remark 4.5
  • Proposition 4.6: Orthogonality of the Shift in Output
  • Proposition 4.7: Near-Orthogonality of the Shift in $\mathbf{X}^l$ to the Principal Component of $\mathbf{X}^l$
  • Remark 4.8
  • Proposition 4.9: Accumulated Shift Orthogonality in the Final Output
  • Remark 4.10
  • Definition E.1: Performance Degradation
  • Definition E.2: Knowledge Retention
  • ...and 14 more