Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continuous Subspace Optimization for Continual Learning

Quan Cheng, Yuanyu Wan, Lingyu Wu, Chenping Hou, Lijun Zhang

TL;DR

This work targets catastrophic forgetting in continual learning by fine-tuning pre-trained Vision Transformers through a sequence of gradient-derived subspaces. By enforcing orthogonality between the current task's optimization subspace and the accumulated historical subspace, and by consolidating task-specific directions via Frequent Directions, CoSO enables memory-efficient, progressive learning across many tasks. Empirical results on ImageNet-R, CIFAR100, and DomainNet show CoSO consistently surpassing state-of-the-art PEFT baselines, with notable gains in long task sequences and robustness across diverse domains. The approach offers a scalable pathway for continual adaptation of foundation models and points to future extensions in multi-modal settings.

Abstract

Continual learning aims to learn multiple tasks sequentially while preserving prior knowledge, but faces the challenge of catastrophic forgetting when adapting to new tasks. Recently, approaches leveraging pre-trained models have gained increasing popularity in mitigating this issue, due to the strong generalization ability of foundation models. To adjust pre-trained models for new tasks, existing methods usually employ low-rank adaptation, which restricts parameter updates to a fixed low-rank subspace. However, constraining the optimization space inherently compromises the model's learning capacity, resulting in inferior performance. To address this limitation, we propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the model in a series of subspaces rather than a single one. These sequential subspaces are dynamically determined through the singular value decomposition of the gradients. CoSO updates the model by projecting gradients onto these subspaces, ensuring memory-efficient optimization. To mitigate forgetting, the optimization subspace of each task is constrained to be orthogonal to the historical task subspace. During task learning, CoSO maintains a task-specific component that captures the critical update directions for the current task. Upon completing a task, this component is used to update the historical task subspace, laying the groundwork for subsequent learning. Extensive experiments on multiple datasets demonstrate that CoSO significantly outperforms state-of-the-art methods, especially in challenging scenarios with long task sequences.

Continuous Subspace Optimization for Continual Learning

TL;DR

This work targets catastrophic forgetting in continual learning by fine-tuning pre-trained Vision Transformers through a sequence of gradient-derived subspaces. By enforcing orthogonality between the current task's optimization subspace and the accumulated historical subspace, and by consolidating task-specific directions via Frequent Directions, CoSO enables memory-efficient, progressive learning across many tasks. Empirical results on ImageNet-R, CIFAR100, and DomainNet show CoSO consistently surpassing state-of-the-art PEFT baselines, with notable gains in long task sequences and robustness across diverse domains. The approach offers a scalable pathway for continual adaptation of foundation models and points to future extensions in multi-modal settings.

Abstract

Continual learning aims to learn multiple tasks sequentially while preserving prior knowledge, but faces the challenge of catastrophic forgetting when adapting to new tasks. Recently, approaches leveraging pre-trained models have gained increasing popularity in mitigating this issue, due to the strong generalization ability of foundation models. To adjust pre-trained models for new tasks, existing methods usually employ low-rank adaptation, which restricts parameter updates to a fixed low-rank subspace. However, constraining the optimization space inherently compromises the model's learning capacity, resulting in inferior performance. To address this limitation, we propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the model in a series of subspaces rather than a single one. These sequential subspaces are dynamically determined through the singular value decomposition of the gradients. CoSO updates the model by projecting gradients onto these subspaces, ensuring memory-efficient optimization. To mitigate forgetting, the optimization subspace of each task is constrained to be orthogonal to the historical task subspace. During task learning, CoSO maintains a task-specific component that captures the critical update directions for the current task. Upon completing a task, this component is used to update the historical task subspace, laying the groundwork for subsequent learning. Extensive experiments on multiple datasets demonstrate that CoSO significantly outperforms state-of-the-art methods, especially in challenging scenarios with long task sequences.
Paper Structure (17 sections, 1 theorem, 19 equations, 2 figures, 6 tables, 1 algorithm)

This paper contains 17 sections, 1 theorem, 19 equations, 2 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Continual Learning
  4. Low-rank Optimization in Offline Learning
  5. Methodology
  6. Preliminaries
  7. Continuous Subspaces Optimization
  8. Historical Task Space Update
  9. Experiments
  10. Experimental Settings
  11. Experimental Results
  12. Conclusion
  13. Proof of Proposition \ref{['thm:low_rank2']}
  14. CoSO Algorithm
  15. Experimental Setups and Implementation Details
  16. ...and 2 more sections

Key Result

Proposition 1

Given a sequence of projected gradients $\{G'_{\tau,t}\}_{t=1}^{T}$ and low-rank matrix $\{Q_{\tau,t}\}_{t=1}^{T}$, where $G_{\tau,t}^\prime\in \mathbb{R}^{m\times n}$ and $Q_{\tau,t}\in \mathbb{R}^{m\times r_2}$. The final sketch matrix is $S_{\tau, T}\in\mathbb{R}^{m\times r_2}$. Let $A=\sum_{t=1} where $\sigma_t$ is the $(r_2+1)$-th singular value of $G_{\tau,t}^\prime$ and $[\tilde{A}]_k$ is t

Figures (2)

  • Figure 1: CoSO optimizes the parameters in continual low-rank subspaces, enhancing the learning capacity of models. To mitigate forgetting, the optimization subspaces of the current task are set to be orthogonal to the historical task subspace. While learning a task, CoSO consolidates the low-rank approximation matrices $\{Q_{\tau,t}\}_{t=1}^{T}$ into a task-specific component $S_{\tau,T}$ through Frequent Directions. The dedicated component is then used to update the historical task subspace spanned by $\mathcal{M}_{\tau-1}$.
  • Figure 2: The detailed performance during the learning of ImageNet-R on (a) 5 tasks, (b) 10 tasks, and (c) 20 tasks.

Theorems & Definitions (2)

  • Proposition 1
  • Remark