Sparse Tuning Enhances Plasticity in PTM-based Continual Learning

TL;DR

This work addresses the plasticity–stability trade-off in PTM-based continual learning by identifying that freezing PTMs constrains adaptation, while full fine-tuning risks destroying pre-trained knowledge. It introduces Mutual Information-guided Sparse Tuning (MIST), a plug-and-play pre-adaptation step that selects the top $k=5\%$ MI-sensitive parameters using an MI-based Fisher score and updates only a tiny subset of parameters via a supervised InfoNCE loss $\mathcal{L}_{MI}$ with strong gradient dropout (up to $90\%$) to update about $0.5\%$ of parameters per batch. MI-based gradients inherently impose less interference with the pre-trained feature space than cross-entropy gradients, and diverse batches further stabilize optimization. Empirically, integrating MIST into six freeze-based continual learning methods yields consistent performance gains across five benchmarks, especially under large distribution shifts, with notable improvements on challenging datasets like Cars-196 and ImageNet-R. The approach is lightweight, widely compatible, and accompanied by open-source code, offering a practical strategy to enhance PTM plasticity without sacrificing generalization.

Abstract

Continual Learning with Pre-trained Models holds great promise for efficient adaptation across sequential tasks. However, most existing approaches freeze PTMs and rely on auxiliary modules like prompts or adapters, limiting model plasticity and leading to suboptimal generalization when facing significant distribution shifts. While full fine-tuning can improve adaptability, it risks disrupting crucial pre-trained knowledge. In this paper, we propose Mutual Information-guided Sparse Tuning (MIST), a plug-and-play method that selectively updates a small subset of PTM parameters, less than 5%, based on sensitivity to mutual information objectives. MIST enables effective task-specific adaptation while preserving generalization. To further reduce interference, we introduce strong sparsity regularization by randomly dropping gradients during tuning, resulting in fewer than 0.5% of parameters being updated per step. Applied before standard freeze-based methods, MIST consistently boosts performance across diverse continual learning benchmarks. Experiments show that integrating our method into multiple baselines yields significant performance gains. Our code is available at https://github.com/zhwhu/MIST.

Figures (7)

• Figure 1: (a) indicates learnable parameters, while denotes frozen parameters. MIST leverages MI for pre-adaptation, enabling it to be plugged into other methods. (b) Comparison of different methods in terms of average accuracy, new task accuracy, and additional parameters. MIST achieves superior accuracy through MI-guided sparse tuning.
• Figure 2: Zero-shot accuracy using different sparse update strategies. After training on each ImageNet-R task, we freeze the PTM and train only on a new classification head on the full Cars dataset. Higher accuracy reflects better generalization capability of the PTM learned from each task.
• Figure 3: New task accuracy.
• Figure 4: Incremental accuracy.
• Figure 5: Performance comparison of incremental learning methods with and without the proposed MIST. The inclusion of MIST consistently improves final accuracy across all evaluated methods and datasets.