Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Continual Learning with Bi-Level Routing Mixture-of-Experts

Meng Lou, Yunxiang Fu, Yizhou Yu

TL;DR

Class-incremental learning on very long task sequences remains challenging due to catastrophic forgetting and representation drift in pre-trained models. CaRE introduces Bi-Level Routing Mixture-of-Experts (BR-MoE) to inject discriminative and comprehensive representations at every layer through dynamic router and expert selection, plus a shared EMA adapter for cross-task knowledge. A new OmniBenchmark-V2–based long-sequence evaluation protocol enables scalable benchmarking up to hundreds of tasks. Empirical results show CaRE outperforms strong PTM-based baselines across both long and short sequences, establishing scalability to 300+ tasks and providing a practical benchmark for future continual learning research.

Abstract

Continual learning, especially class-incremental learning (CIL), on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years. However, how to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem. We propose CaRE, a scalable {C}ontinual Le{a}rner with efficient Bi-Level {R}outing Mixture-of-{E}xperts (BR-MoE). The core idea of BR-MoE is a bi-level routing mechanism: a router selection stage that dynamically activates relevant task-specific routers, followed by an expert routing phase that dynamically activates and aggregates experts, aiming to inject discriminative and comprehensive representations into every intermediate network layer. On the other hand, we introduce a challenging evaluation protocol for comprehensively assessing CIL methods across very long task sequences spanning hundreds of tasks. Extensive experiments show that CaRE demonstrates leading performance across a variety of datasets and task settings, including commonly used CIL datasets with classical CIL settings (e.g., 5-20 tasks). To the best of our knowledge, CaRE is the first continual learner that scales to very long task sequences (ranging from 100 to over 300 non-overlapping tasks), while outperforming all baselines by a large margin on such task sequences. Code will be publicly released at https://github.com/LMMMEng/CaRE.git.

Scaling Continual Learning with Bi-Level Routing Mixture-of-Experts

TL;DR

Class-incremental learning on very long task sequences remains challenging due to catastrophic forgetting and representation drift in pre-trained models. CaRE introduces Bi-Level Routing Mixture-of-Experts (BR-MoE) to inject discriminative and comprehensive representations at every layer through dynamic router and expert selection, plus a shared EMA adapter for cross-task knowledge. A new OmniBenchmark-V2–based long-sequence evaluation protocol enables scalable benchmarking up to hundreds of tasks. Empirical results show CaRE outperforms strong PTM-based baselines across both long and short sequences, establishing scalability to 300+ tasks and providing a practical benchmark for future continual learning research.

Abstract

Continual learning, especially class-incremental learning (CIL), on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years. However, how to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem. We propose CaRE, a scalable {C}ontinual Le{a}rner with efficient Bi-Level {R}outing Mixture-of-{E}xperts (BR-MoE). The core idea of BR-MoE is a bi-level routing mechanism: a router selection stage that dynamically activates relevant task-specific routers, followed by an expert routing phase that dynamically activates and aggregates experts, aiming to inject discriminative and comprehensive representations into every intermediate network layer. On the other hand, we introduce a challenging evaluation protocol for comprehensively assessing CIL methods across very long task sequences spanning hundreds of tasks. Extensive experiments show that CaRE demonstrates leading performance across a variety of datasets and task settings, including commonly used CIL datasets with classical CIL settings (e.g., 5-20 tasks). To the best of our knowledge, CaRE is the first continual learner that scales to very long task sequences (ranging from 100 to over 300 non-overlapping tasks), while outperforming all baselines by a large margin on such task sequences. Code will be publicly released at https://github.com/LMMMEng/CaRE.git.
Paper Structure (15 sections, 9 equations, 5 figures, 12 tables)

This paper contains 15 sections, 9 equations, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminaries
  5. Overall Architecture
  6. Bi-Level Routing Mixture-of-Experts
  7. Discussions of BR-MoE
  8. Long Task Sequence Evaluation Protocol
  9. Experiments
  10. Long Task Sequence Evaluations
  11. Short Task Sequence Evaluations
  12. Ablation Studies
  13. Analytical Experiments
  14. Limitations
  15. Conclusion

Figures (5)

  • Figure 1: Incremental performance comparisons between our CaRE and other representative PTM-based CIL methods on the long-sequence evaluation protocol using OmniBenchmark-V2 dataset. Our method outperforms other baselines by a large margin across a variety of settings. "B-$\mathcal{M}$ Inc-$\mathcal{N}$" denotes the number of base classes ($\mathcal{M}$) and the number of incremental classes ($\mathcal{N}$) per task.
  • Figure 2: The workflow of the proposed BR-MoE. (a) The network building block equipped with our BR-MoE. (b) Training and (c) inference pipelines of BR-MoE.
  • Figure 3: Comparison of computational efficiency among different methods.
  • Figure 4: Visualization of our bi-level routing mechanism.
  • Figure 5: Visualization of bi-level router and expert activation patterns. (a) Dynamic router selection (first level) patterns using class perceptrons. (b) Dynamic adapter/expert selection patterns (second level). Rows and columns of each heatmap correspond to router/expert indices and task indices, respectively.