Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning

Ruiqi Liu, Boyu Diao, Zijia An, Zhulin An, Fei Wang, Yongjun Xu

TL;DR

Dynamical Adapter Fusion (DAF) presents a theoretically grounded approach to class-incremental learning by fusing task-specific adapters, the prior global adapter, and a robust initialization into a single fixed global adapter. Grounded in PAC-Bayes bounds, it derives optimal fusion coefficients via a Taylor-expanded loss and Lagrange multipliers, balancing stability and plasticity without storing historical adapters. A robust initialization built from historical adapters anchors learning to global knowledge, while a Fisher-based Hessian approximation enables efficient, dynamic fusion. Empirical results across CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet show state-of-the-art exemplar-free performance with high inference throughput, demonstrating the practical value for resource-constrained continual learning scenarios.

Abstract

Class-Incremental Learning (CIL) requires models to continuously acquire new classes without forgetting previously learned ones. A dominant paradigm involves freezing a pre-trained model and training lightweight, task-specific adapters. However, maintaining task-specific parameters hinders knowledge transfer and incurs high retrieval costs, while naive parameter fusion often leads to destructive interference and catastrophic forgetting. To address these challenges, we propose Dynamical Adapter Fusion (DAF) to construct a single robust global adapter. Grounded in the PAC-Bayes theorem, we derive a fusion mechanism that explicitly integrates three components: the optimized task-specific adapter parameters, the previous global adapter parameters, and the initialization parameters. We utilize the Taylor expansion of the loss function to derive the optimal fusion coefficients, dynamically achieving the best balance between stability and plasticity. Furthermore, we propose a Robust Initialization strategy to effectively capture global knowledge patterns. Experiments on multiple CIL benchmarks demonstrate that DAF achieves state-of-the-art (SOTA) performance.

Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning

TL;DR

Dynamical Adapter Fusion (DAF) presents a theoretically grounded approach to class-incremental learning by fusing task-specific adapters, the prior global adapter, and a robust initialization into a single fixed global adapter. Grounded in PAC-Bayes bounds, it derives optimal fusion coefficients via a Taylor-expanded loss and Lagrange multipliers, balancing stability and plasticity without storing historical adapters. A robust initialization built from historical adapters anchors learning to global knowledge, while a Fisher-based Hessian approximation enables efficient, dynamic fusion. Empirical results across CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet show state-of-the-art exemplar-free performance with high inference throughput, demonstrating the practical value for resource-constrained continual learning scenarios.

Abstract

Class-Incremental Learning (CIL) requires models to continuously acquire new classes without forgetting previously learned ones. A dominant paradigm involves freezing a pre-trained model and training lightweight, task-specific adapters. However, maintaining task-specific parameters hinders knowledge transfer and incurs high retrieval costs, while naive parameter fusion often leads to destructive interference and catastrophic forgetting. To address these challenges, we propose Dynamical Adapter Fusion (DAF) to construct a single robust global adapter. Grounded in the PAC-Bayes theorem, we derive a fusion mechanism that explicitly integrates three components: the optimized task-specific adapter parameters, the previous global adapter parameters, and the initialization parameters. We utilize the Taylor expansion of the loss function to derive the optimal fusion coefficients, dynamically achieving the best balance between stability and plasticity. Furthermore, we propose a Robust Initialization strategy to effectively capture global knowledge patterns. Experiments on multiple CIL benchmarks demonstrate that DAF achieves state-of-the-art (SOTA) performance.
Paper Structure (36 sections, 3 theorems, 44 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 36 sections, 3 theorems, 44 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Training from Scratch
  4. Pre-Trained Model-based CIL
  5. Methodology
  6. Preliminaries
  7. Class-Incremental Learning.
  8. PTM-based CIL via Task-Specific Adapters.
  9. Theoretical Analysis
  10. Dynamical Adapter Fusion Method
  11. Optimization Approximation
  12. Robust Initialization via Historical Knowledge
  13. Overview of Our Method
  14. Experiments
  15. Implementation Details
  16. ...and 21 more sections

Key Result

Theorem 3.1

Let $\mathcal{D}_t=(x_1,...,x_{N_t})$ be an iid set sampled from distribution $\mu_t$ for task $t$. Let prior $P_t = P_{\text{init}} \otimes Q_{t-1}^g$ and posterior $Q_t = Q_t^s \otimes Q_t^g$ be defined on $\mathcal{H}$. For any loss $\ell$ bounded by $K$, $\lambda>0$, and $\delta\in [0,1]$, the f

Figures (7)

  • Figure 1: Efficiency and performance evaluation on CIFAR-100. We plot Average Accuracy ($\bar{\mathcal{A}}$) against Training Throughput (samples/s). The color of each point represents the Inference Throughput (samples/ms). The ideal method occupies the top-right corner with a dark red color (highest inference throughput). TUNA wang2025integrating is evaluated here using its single global adapter. Our DAF achieves this optimal balance, significantly outperforming complex methods in both speed and precision.
  • Figure 2: Overview of the proposed DAF framework. The pipeline consists of four distinct phases: (1) Initialization: We construct a Robust Initialization$\theta_p$ via a recursive update of implicit past adapters; (2) Current Task Training: The Optimized Task Adapter$\theta_t$ is learned on new data, and statistics (Gradients, Fisher Matrix) are computed; (3) Dynamical Adapter Fusion : This core module operates in two steps: (a) the $\beta_t$ Calculator derives the optimal coefficient using the Lagrangian solution, and (b) the Fusion process integrates the Robust Initialization $\theta_p$, the Previous Global Adapter $\theta_{t-1}^*$, and the Optimized Task Adapter $\theta_t$ to form the updated model; (4) Inference: The resulting Single Global Adapter $\theta_t^*$ performs efficient, retrieval-free inference on all learned classes.
  • Figure 3: Incremental accuracy trends on four benchmarks using the ViT-B/16-IN1K backbone. DAF (red curve) is compared against state-of-the-art exemplar-free methods. Across all datasets, DAF consistently demonstrates superior performance. The curves highlight our method's robust ability to mitigate catastrophic forgetting and maintain high accuracy as the number of tasks increases.
  • Figure 4: Comparison with rehearsal-based methods. Rehearsal-based methods use 20 exemplars per class. Using ViT-B/16-IN21K as the backbone, DAF significantly outperforms strong replay baselines (iCaRL, MEMO, FOSTER) across CIFAR-100, ImageNet-R, and ObjectNet. Notably, DAF achieves this without storing past exemplars, validating the effectiveness of our parameter fusion mechanism over traditional sample replay.
  • Figure 5: Ablation and sensitivity analysis of DAF on ImageNet-R using the ViT-B/16-IN21K backbone. (a) Incremental ablation study demonstrating the cumulative gains from Static Fusion, Dynamical Fusion, and Robust Initialization. (b) Dual-axis analysis of hyperparameter $\alpha$, illustrating the explicit trade-off between Stability (Green) and Plasticity (Red). (c) Evaluation of the balancing weight $\gamma$, confirming that the theoretically derived equal weighting ($\gamma=0.5$) of historical priors yields optimal performance.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Theorem 3.1
  • Lemma 3.1: Adapted from alquier2016properties, Thm 4.1
  • Definition 3.2: Factorized Parameter Space and Distributions
  • Theorem 3.3
  • proof
  • Remark 3.4