ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder

TL;DR

ProCLIP tackles the limitations of CLIP in processing long and multilingual text by introducing a Progressive vision-language Alignment framework that couples an LLM-based text embedder with the CLIP image encoder through a two-stage curriculum. In Stage 1, cross-architecture knowledge distillation transfers CLIP's textual knowledge to the LLM embedder, establishing initial alignment with the image encoder via instance semantic and embedding-structure losses. In Stage 2, standard image–text contrastive learning is performed with self-distillation regularization to prevent forgetting and preserve pretrained knowledge, guided by an InfoNCE objective. Across multiple data scales and model architectures, ProCLIP consistently improves zero-shot classification, cross-modal retrieval (including long-text and multilingual cases), robustness to distribution shifts, and fine-grained understanding, demonstrating the practicality of progressive, distillation-guided cross-modal alignment. The approach offers a generalizable pathway to integrate powerful LLM-based text representations into vision-language models without sacrificing pretrained generalization, supporting broader real-world applicability.

Abstract

The original CLIP text encoder is limited by a maximum input length of 77 tokens, which hampers its ability to effectively process long texts and perform fine-grained semantic understanding. In addition, the CLIP text encoder lacks support for multilingual inputs. All these limitations significantly restrict its applicability across a broader range of tasks. Recent studies have attempted to replace the CLIP text encoder with an LLM-based embedder to enhance its ability in processing long texts, multilingual understanding, and fine-grained semantic comprehension. However, because the representation spaces of LLMs and the vision-language space of CLIP are pretrained independently without alignment priors, direct alignment using contrastive learning can disrupt the intrinsic vision-language alignment in the CLIP image encoder, leading to an underutilization of the knowledge acquired during pre-training. To address this challenge, we propose ProCLIP, a curriculum learning-based progressive vision-language alignment framework to effectively align the CLIP image encoder with an LLM-based embedder. Specifically, ProCLIP first distills knowledge from CLIP's text encoder into the LLM-based embedder to leverage CLIP's rich pretrained knowledge while establishing initial alignment between the LLM embedder and CLIP image encoder. Subsequently, ProCLIP further aligns the CLIP image encoder with the LLM-based embedder through image-text contrastive tuning, employing self-distillation regularization to avoid overfitting. To achieve a more effective alignment, instance semantic alignment loss and embedding structure alignment loss are employed during representation inheritance and contrastive tuning. The Code is available at https://github.com/VisionXLab/ProCLIP.

Figures (7)

• Figure 1: Illustration of research gap. Previous work directly aligns the LLM-based embedder with the CLIP image encoder, disregarding the valuable knowledge embedded in the pre-trained CLIP model. In contrast, ProCLIP first transfers knowledge from CLIP’s text encoder to the LLM embedder via distillation, establishing an initial alignment. It then refines the alignment between the CLIP image encoder and the LLM-based embedder through image-text contrastive learning with self-distillation regularization.
• Figure 2: The training pipeline of our proposed ProCLIP. It consists of representation inheritance via cross-architecture distillation and contrastive tuning integrated with self-distillation regularization.
• Figure 3: Per-language image-text retrieval performance on the XM3600 benchmark.
• Figure 4: MMVP performance comparison. ProCLIP presents excellent performance.
• Figure 5: Comparison with other methods across different model scales and LLM embedders.