A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

TL;DR

This work tackles the multilingual extension and edge deployment of vision-language models by proposing DC-CLIP, a two-stage compression framework that distills image and text encoders from the AltCLIP teacher into lightweight bilingual students and then aligns their cross-modal representations using a contrastive loss. The first stage uses feature-map distillation with a 1×1 adapter and SmoothL1 losses to transfer rich visual and multilingual textual knowledge, while the second stage applies InfoNCE-based cross-modal alignment with a loss $Loss_{CL} = \tfrac{1}{2}(\mathcal{L}_{I-T} + \mathcal{L}_{T-I})$ to improve multilingual fusion. The approach demonstrates strong English zero-shot performance and competitive Chinese results with limited data, comparing favorably to baselines of similar parameter scale, and highlights the effectiveness of feature-map targets over logits for cross-lingual distillation. The work offers a practical route to deploy efficient bilingual V-L models on mobile/edge devices, while also identifying data-quality challenges in Chinese annotations and suggesting future expansion to additional languages and domains.

Abstract

Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focused on this problem and proposed improved models, such as CN-CLIP and AltCLIP, to facilitate their applicability to Chinese and even other languages. Nevertheless, these models suffer from high latency and a large memory footprint in inference, which limits their further deployment on resource-constrained edge devices. In this work, we propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. During the first stage, lightweight image/text student models are designed to learn robust visual/multilingual textual feature representation ability from corresponding teacher models, respectively. Subsequently, the multilingual vision-language alignment stage enables effective alignment of visual and multilingual textual features to further improve the model's multilingual performance. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context and competitive performance in the Chinese context, even with less training data, when compared to existing models of similar parameter magnitude. The evaluation demonstrates the effectiveness of our designed training mechanism.

Figures (1)

• Figure 1: The framework of the proposed method comprises two progressive modules: (a) the vision-language feature distillation module and (b) the vision-language feature alignment module. In the first stage, module (a) extracts key knowledge from the teacher model's image and text encoders and transfers it to the student model. Subsequently, module (b) further aligns the vision-language feature using contrastive learning strategy to enhance the model's performance.