Post-pre-training for Modality Alignment in Vision-Language Foundation Models

TL;DR

This work tackles the modality gap in vision-language CLIP models by introducing CLIP-Refine, a post-pre-training procedure that sits between pre-training and fine-tuning. It combines Random Feature Alignment (RaFA), which pulls image and text features toward a shared prior $p(z)$ (e.g., $\mathcal{N}(0,I)$), with Hybrid Contrastive-Distillation (HyCD), which preserves past knowledge via KL-distillation from a frozen teacher while learning new cross-modal cues through alpha-blended supervision. The method operates on small datasets (e.g., COCO Captions) and requires only one epoch of training, yielding significant improvements in zero-shot classification and cross-modal retrieval without sacrificing prior capabilities. By simultaneously improving modality gap and feature-space uniformity, CLIP-Refine offers a practical, scalable path to better cross-modal alignment applicable to a range of pre-trained backbones, with dataset quality and prior choice playing important roles in performance.

Abstract

Contrastive language image pre-training (CLIP) is an essential component of building modern vision-language foundation models. While CLIP demonstrates remarkable zero-shot performance on downstream tasks, the multi-modal feature spaces still suffer from a modality gap, which is a gap between image and text feature clusters and limits downstream task performance. Although existing works attempt to address the modality gap by modifying pre-training or fine-tuning, they struggle with heavy training costs with large datasets or degradations of zero-shot performance. This paper presents CLIP-Refine, a post-pre-training method for CLIP models at a phase between pre-training and fine-tuning. CLIP-Refine aims to align the feature space with 1 epoch training on small image-text datasets without zero-shot performance degradations. To this end, we introduce two techniques: random feature alignment (RaFA) and hybrid contrastive-distillation (HyCD). RaFA aligns the image and text features to follow a shared prior distribution by minimizing the distance to random reference vectors sampled from the prior. HyCD updates the model with hybrid soft labels generated by combining ground-truth image-text pair labels and outputs from the pre-trained CLIP model. This contributes to achieving both maintaining the past knowledge and learning new knowledge to align features. Our extensive experiments with multiple classification and retrieval tasks show that CLIP-Refine succeeds in mitigating the modality gap and improving the zero-shot performance.

Figures (8)

• Figure 1: Scope of our work: post-pre-training to align the modality gap in pre-trained CLIP models. We aim to address the modality gap and enhance the generalization performance of pre-trained CLIP models through lightweight training.
• Figure 2: Intuition of CLIP-Refine (proposed method). CLIP-Refine modifies a pre-trained CLIP model by Random Feature Alignment (RaFA, left) and Hybrid Contrastive-Distillation (HyCD, right). RaFA penalizes image and text features to follow a shared prior distribution by minimizing the gap between the features and a random reference vector sampled from the prior. HyCD trains the model with the KL-divergence-based distillation loss using hybrid soft labels composed by alpha blending the ground truth label of image-text pairs with the output of the pre-trained model (teacher).
• Figure 3: PCA visualization of multi-modal feature spaces of trained models (Flickr8K, ViT-B/32). CLIP-Refine reduces the modality gap while keeping the form of feature distributions for each modality.
• Figure 8: Zero-shot classification accuracy averaged on 12 datasets when varying mini-batch sizes in post-pre-training.
• Figure IX: Zero-shot classification accuracy averaged on 12 datasets when varying balancing parameters between $\mathcal{L}_\mathrm{RaFA}$ and $\mathcal{L}_\mathrm{HyCD}$ (ViT-B/32).