Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning

TL;DR

This work proposes a Multi-Stage Knowledge Integration network (MulKI) to emulate the human learning process in distillation methods, and demonstrates significant improvements in maintaining zero-shot capabilities while supporting continual learning across diverse downstream tasks, showcasing its potential in adapting VLMs to evolving data distributions.

Abstract

Vision Language Models (VLMs), pre-trained on large-scale image-text datasets, enable zero-shot predictions for unseen data but may underperform on specific unseen tasks. Continual learning (CL) can help VLMs effectively adapt to new data distributions without joint training, but faces challenges of catastrophic forgetting and generalization forgetting. Although significant progress has been achieved by distillation-based methods, they exhibit two severe limitations. One is the popularly adopted single-teacher paradigm fails to impart comprehensive knowledge, The other is the existing methods inadequately leverage the multimodal information in the original training dataset, instead they rely on additional data for distillation, which increases computational and storage overhead. To mitigate both limitations, by drawing on Knowledge Integration Theory (KIT), we propose a Multi-Stage Knowledge Integration network (MulKI) to emulate the human learning process in distillation methods. MulKI achieves this through four stages, including Eliciting Ideas, Adding New Ideas, Distinguishing Ideas, and Making Connections. During the four stages, we first leverage prototypes to align across modalities, eliciting cross-modal knowledge, then adding new knowledge by constructing fine-grained intra- and inter-modality relationships with prototypes. After that, knowledge from two teacher models is adaptively distinguished and re-weighted. Finally, we connect between models from intra- and inter-task, integrating preceding and new knowledge. Our method demonstrates significant improvements in maintaining zero-shot capabilities while supporting continual learning across diverse downstream tasks, showcasing its potential in adapting VLMs to evolving data distributions.

Figures (6)

• Figure 1: Performance (%) changes during continual learning on MTILzscl for each task, in which MulKI with $C_0$ and with $C_{i-1}$ represent adopting only the initial model and only the preceding model for distillation, respectively. The metric "Transfer" denotes averaging the results of unseen tasks only, while "Current Avg." averages only the results of seen tasks, the two metrics represent the mitigation of two forgetting issues, respectively. Our method effectively mitigates both forgetting issues. We omit the results of task 0 (lower than 55%) in (b) for clearer observation. We exclude LwF-VR here for its low accuracy.
• Figure 2: Illustration of three distillation paradigms and the utilization of additional data, where additional data is marked in bold. (a) LwF-VRlwfvr utilizes texts from past tasks (including texts in pretrain) to implement single-teacher distillation between current and previous models. (b) ZSCLzscl utilizes reference images and texts to implement single-teacher distillation between current and the initial models. (c) Our method utilizes only training data of current task, and implementing dual-teacher distillation among current, previous and the initial models.
• Figure 3: Overview of the proposed MulKI network. It utilizes only images and texts from current task to implement dual-teacher distillation. Prototypes are updated with current image features. Then multi-level knowledge distillation is employed on cross-modal relations, which is acquired under the guidance of prototypes. The final multi-level distillation loss is computed by combining the loss from two teacher models.
• Figure 4: The impact of the upperbound of $\gamma$ and its increasing step for CIFAR-100 on 10-step setting. We report the "Last" metric. The chosen hyper-parameters are marked with dash lines. (a) The impact of $\Gamma$ when $\Delta \gamma$ is set as 0.04. (b) The impact of $\Delta \gamma$ when $\Gamma$ is set as 0.98.
• Figure 5: t-SNE visualization of features of (a) CLIP, (b) ZSCL, (c) MoE, (d) Few-shot MulKI and (e) Full-Shot MulKI for MTIL benchmark under Order I. ZSCL and our MulKI under few-shot setting maintain the similar feature distribution to the original CLIP, while MoE significantly distorts the feature space. 0-10 denote the task ids in Order I.