Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning

JinLi He, Liang Bai, Xian Yang

TL;DR

This work characterize model's forgetting from the perspective of parameter update magnitude and formalize it as knowledge degradation induced by task-specific drift in the parameter space, which has not been fully captured in previous studies due to their assumption of a unified parameter space.

Abstract

The magnitude of parameter updates are considered a key factor in continual learning. However, most existing studies focus on designing diverse update strategies, while a theoretical understanding of the underlying mechanisms remains limited. Therefore, we characterize model's forgetting from the perspective of parameter update magnitude and formalize it as knowledge degradation induced by task-specific drift in the parameter space, which has not been fully captured in previous studies due to their assumption of a unified parameter space. By deriving the optimal parameter update magnitude that minimizes forgetting, we unify two representative update paradigms, frozen training and initialized training, within an optimization framework for constrained parameter updates. Our theoretical results further reveals that sequence tasks with small parameter distances exhibit better generalization and less forgetting under frozen training rather than initialized training. These theoretical insights inspire a novel hybrid parameter update strategy that adaptively adjusts update magnitude based on gradient directions. Experiments on deep neural networks demonstrate that this hybrid approach outperforms standard training strategies, providing new theoretical perspectives and practical inspiration for designing efficient and scalable continual learning algorithms.

Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning

TL;DR

This work characterize model's forgetting from the perspective of parameter update magnitude and formalize it as knowledge degradation induced by task-specific drift in the parameter space, which has not been fully captured in previous studies due to their assumption of a unified parameter space.

Abstract

The magnitude of parameter updates are considered a key factor in continual learning. However, most existing studies focus on designing diverse update strategies, while a theoretical understanding of the underlying mechanisms remains limited. Therefore, we characterize model's forgetting from the perspective of parameter update magnitude and formalize it as knowledge degradation induced by task-specific drift in the parameter space, which has not been fully captured in previous studies due to their assumption of a unified parameter space. By deriving the optimal parameter update magnitude that minimizes forgetting, we unify two representative update paradigms, frozen training and initialized training, within an optimization framework for constrained parameter updates. Our theoretical results further reveals that sequence tasks with small parameter distances exhibit better generalization and less forgetting under frozen training rather than initialized training. These theoretical insights inspire a novel hybrid parameter update strategy that adaptively adjusts update magnitude based on gradient directions. Experiments on deep neural networks demonstrate that this hybrid approach outperforms standard training strategies, providing new theoretical perspectives and practical inspiration for designing efficient and scalable continual learning algorithms.
Paper Structure (9 sections, 7 theorems, 20 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 9 sections, 7 theorems, 20 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Previous Research
  3. Problem Setup
  4. Frozen Training vs Initialized Training
  5. Implications on Practical CL
  6. Hybrid Algorithm Framework
  7. Datasets and Evaluation
  8. Experimental Analysis
  9. Conclusion

Key Result

Lemma 4.2

Under frozen training, define $r = 1 - \frac{n_{(1)}}{p+p_{(1)}}$, $r_{A} = 1 - \frac{n_{(2)}}{p_{(2)}}$. After completing $(\tilde{u}_{(2)},\tilde{q}_{(2)})$, the average deviation from the previous task's parameters is given by: where $v_{(1)}= 1 + \frac{p\,n_{(2)}}{(p+p_{(1)})(p_{(2)}-n_{(2)}-1)} - \frac{p_{(1)}n_{(2)}}{(p+p_{(1)})p_{(2)}}$.

Figures (2)

  • Figure 1: Switching tasks shifts target parameter space, causing forgetting in continual learning. The red symbols shows that a shared space enables fast adaptation and knowledge retention.
  • Figure 2: Frozen training and initialized training. Initialized training updates all parameters during training, whereas frozen training preserves general knowledge by fixing most parameters and adapts to new tasks using a few trainable parameters.

Theorems & Definitions (9)

  • Remark 4.1
  • Lemma 4.2
  • Lemma 4.3
  • Theorem 4.4
  • Lemma 4.5
  • Theorem 4.6
  • Proposition 4.7
  • Remark 4.8
  • Corollary 4.9