Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning

TL;DR

SECA tackles the stability-plasticity dilemma in rehearsal-free class-incremental learning by leveraging textual semantic priors from CLIP. It introduces SG-AKT to perform instance-adaptive knowledge transfer via a fixed adapter pool and SE-VPR to inject inter-class textual semantics into visual prototypes, achieving a robust hybrid classifier. Empirical results on ImageNetR, ImageNetA, and CIFAR100 show SECA and its replay-enhanced variant SECA++ surpass state-of-the-art PEFT-based continual learning methods, with notable gains in Last accuracy. The framework demonstrates that text-grounded semantic priors can improve both forgetting resistance and plasticity, offering practical benefits for open-world, multi-modal continual learning scenarios.

Abstract

Continual learning (CL) aims to equip models with the ability to learn from a stream of tasks without forgetting previous knowledge. With the progress of vision-language models like Contrastive Language-Image Pre-training (CLIP), their promise for CL has attracted increasing attention due to their strong generalizability. However, the potential of rich textual semantic priors in CLIP in addressing the stability-plasticity dilemma remains underexplored. During backbone training, most approaches transfer past knowledge without considering semantic relevance, leading to interference from unrelated tasks that disrupt the balance between stability and plasticity. Besides, while text-based classifiers provide strong generalization, they suffer from limited plasticity due to the inherent modality gap in CLIP. Visual classifiers help bridge this gap, but their prototypes lack rich and precise semantics. To address these challenges, we propose Semantic-Enriched Continual Adaptation (SECA), a unified framework that harnesses the anti-forgetting and structured nature of textual priors to guide semantic-aware knowledge transfer in the backbone and reinforce the semantic structure of the visual classifier. Specifically, a Semantic-Guided Adaptive Knowledge Transfer (SG-AKT) module is proposed to assess new images' relevance to diverse historical visual knowledge via textual cues, and aggregate relevant knowledge in an instance-adaptive manner as distillation signals. Moreover, a Semantic-Enhanced Visual Prototype Refinement (SE-VPR) module is introduced to refine visual prototypes using inter-class semantic relations captured in class-wise textual embeddings. Extensive experiments on multiple benchmarks validate the effectiveness of our approach.

Figures (3)

• Figure 1: (a) Existing methods exhibit limited stability-plasticity trade-offs due to the unrelated knowledge transfer in backbone and the modality gap in classifier. (b) Our SECA leverage textual priors to (1) prioritize transferring relevant knowledge (Dog classes) when learning new classes (Cat), and (2) inject semantic relations into the visual classifier to bridge the modality gap, improving this trade-off.
• Figure 2: Overall framework. Our SECA is composed of two novel components: (a) The SG-AKT module that utilizes textual semantic vectors to aggregates relevant visual representations from a pool of historical adapters for distillation; (b) The SE-VPR module that leverages inter-class semantic relationships to refine CLIP prototypes, constructing a powerful visual-side classifier.
• Figure 3: Model performances under different pool sizes $\vert \mathcal{P} \vert$ and temperatures $\tau'$.