Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning

TL;DR

The paper tackles catastrophic forgetting in continual learning by integrating model merging into the training-inference loop through Perturb-and-Merge (P&M). After each task, P&M trains a task-specific optimum and merges it with the previous inference parameters using a closed-form coefficient $\alpha_t^*$ derived from a second-order loss expansion, with Hessians approximated by the diagonal empirical Fisher. To further stabilize merging, it introduces a perturbation-based regularizer aligned with the task vector, approximated efficiently via symmetric finite differences and randomized perturbations, avoiding extra forward/backward passes. The method is extended with LoRA to enhance memory efficiency, yielding state-of-the-art results across multiple CL benchmarks with modest resource overhead. Overall, P&M demonstrates that a post-training model-merge strategy, guided by a principled optimization of the merge coefficient and robust regularization, can substantially mitigate forgetting while preserving plasticity and generalization.

Abstract

Continual Learning (CL) aims to enable models to continuously acquire new knowledge from a sequence of tasks with avoiding the forgetting of learned information. However, existing CL methods only rely on the parameters of the most recent task for inference, which makes them susceptible to catastrophic forgetting. Inspired by the recent success of model merging techniques, we propose \textbf{Perturb-and-Merge (P\&M)}, a novel continual learning framework that integrates model merging into the CL paradigm to mitigate forgetting. Specifically, after training on each task, P\&M constructs a new model by forming a convex combination of the previous model and the newly trained task-specific model. Through theoretical analysis, We minimize the total loss increase across all tasks and derive a closed-form solution for the merging coefficient under mild assumptions. To further improve the performance of the merged model, we observe that the degradation introduced during merging can be alleviated by a regularization term composed of the task vector and the Hessian matrix of the loss function. Interestingly, we show that this term can be efficiently approximated using second-order symmetric finite differences, and a stochastic perturbation strategy along the task vector direction is accordingly devised which incurs no additional forward or backward passes while providing an effective approximation of the regularization term. Finally, we combine P\&M with LoRA, a parameter-efficient fine-tuning method, to reduce memory overhead. Our proposed approach achieves state-of-the-art performance on several continual learning benchmark datasets. The code is available at https://github.com/qhmiao/P-M-for-Continual-Learning.

Figures (3)

• Figure 1: P&M reduces forgetting with minimal impact on plasticity. Comparison of LoRA and LoRA-P&M across four benchmarks. P&M achieves similar plasticity (current task performance) while significantly mitigating forgetting (average performance drop on previous tasks), resulting in higher overall ACC.
• Figure 2: Loss landscape visualization on ImageNet-R and CUB.M denotes using only Infer after Merging (no perturbation during training). Each subplot shows the average loss surface after merging at Task 4 and Task 7, with axes representing weights $\alpha$ and $\beta$ in the convex combination $\hat{\theta}_t = \beta \hat{\theta}_{t-1} + \alpha \theta_t^*$. The convex path lies in a low-loss region (Obs. ②), and our optimal $\alpha$ consistently locates near the minimum (Obs. ③). Task-vector perturbation further enlarges the flat region (Obs. ④).
• Figure 3: Hyperparameter ablation on ImageNet-A. Left: performance under varying $p_0$ values (probability of zero perturbation). Right: performance under different $\epsilon$ values (magnitude of perturbation).