ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model

TL;DR

ComKD-CLIP addresses the practical need to deploy CLIP-like models in resource-constrained environments by comprehensively distilling knowledge from a large teacher into a compact student. It introduces two key modules, Image Feature Alignment (IFAlign) and Educational Attention (EduAttention), to distill both image-feature extraction strategies and cross-modal fusion behavior during the text-image feature fusion process, with an optional refinement step using teacher fusion results. The method achieves superior or highly competitive performance across 11 datasets in base-to-novel, cross-dataset, and domain-generalization settings, outperforming several state-of-the-art distillation approaches. This work enhances the practicality of CLIP by enabling smaller models to retain high multimodal understanding, which is critical for deployment in limited-resource environments, with a strong emphasis on reproducibility and detailed experimental protocols.

Abstract

Contrastive Language-Image Pre-training (CLIP) models excel in integrating semantic information between images and text through contrastive learning techniques. It has achieved remarkable performance in various multimodal tasks. However, the deployment of large CLIP models is hindered in resource-limited environments, while smaller models frequently fail to meet the performance benchmarks required for practical applications. In this paper, we propose a novel approach, ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model, which aims to comprehensively distill the knowledge from a large teacher CLIP model into a smaller student model, ensuring comparable performance with significantly reduced parameters. ComKD-CLIP is composed of two key mechanisms: Image Feature Alignment (IFAlign) and Educational Attention (EduAttention). IFAlign makes the image features extracted by the student model closely match those extracted by the teacher model, enabling the student to learn teacher's knowledge of extracting image features. EduAttention explores the cross-relationships between text features extracted by the teacher model and image features extracted by the student model, enabling the student model to learn how the teacher model integrates text-image features. In addition, ComKD-CLIP can refine the knowledge distilled from IFAlign and EduAttention by leveraging the text-image feature fusion results of the teacher model, ensuring the student model accurately absorbs the teacher's knowledge. Extensive experiments conducted on 11 datasets have demonstrated the superiority of the proposed method.

Figures (2)

• Figure 1: (a) Schematic of proposed ComKD-CLIP framework. (b) Comparison of the Harmonic Mean (HM) for generalization from base to novel categories. All methods utilize the ViT-B/16 image encoder from the pretrained CLIP model. ComKD-CLIP is proud to achieve the best performance in 8 out of 11 diverse recognition datasets.
• Figure 2: Overview of our proposed ComKD-CLIP framework. (a) Utilization of the cue learning method with a well-trained large teacher CLIP model; (b) A smaller student CLIP model, which is trained with learnable cues and reuses the text features from the teacher model, requiring training only for the image encoder branch; (c) The schematic of IFAlign module; (d) The inference process within the trained student model, where the text encoder branch reuses the text features of the teacher model; (e) The schematic of EduAttention module.