Assessing and Learning Alignment of Unimodal Vision and Language Models

TL;DR

The paper tackles how well unimodal vision and language models align and how to measure and improve this alignment for practical vision-language tasks. It introduces alignment probing to quantify cross-modal alignment with frozen backbones and shows that SSL clustering quality and language understanding strongly influence alignment. Building on these insights, it proposes SAIL, a data- and compute-efficient transfer-learning framework that uses a nonlinear alignment layer, a sigmoid-based contrastive loss, and high-quality captions to align pretrained unimodal models, achieving competitive or superior zero-shot and retrieval performance with only a fraction of the data required by CLIP. SAIL also enhances the language-compatibility of vision encoders, enabling their integration with multimodal LLMs and improving downstream tasks such as VQA, open-vocabulary segmentation, and complex reasoning, all while remaining open-source. These findings offer a practical pathway for resource-efficient foundation VLMs and broaden the accessibility of high-performance vision-language systems.

Abstract

How well are unimodal vision and language models aligned? Although prior work have approached answering this question, their assessment methods do not directly translate to how these models are used in practical vision-language tasks. In this paper, we propose a direct assessment method, inspired by linear probing, to assess vision-language alignment. We identify that the degree of alignment of the SSL vision models depends on their SSL training objective, and we find that the clustering quality of SSL representations has a stronger impact on alignment performance than their linear separability. Next, we introduce Swift Alignment of Image and Language (SAIL), a efficient transfer learning framework that aligns pretrained unimodal vision and language models for downstream vision-language tasks. Since SAIL leverages the strengths of pretrained unimodal models, it requires significantly fewer (6%) paired image-text data for the multimodal alignment compared to models like CLIP which are trained from scratch. SAIL training only requires a single A100 GPU, 5 hours of training and can accommodate a batch size up to 32,768. SAIL achieves 73.4% zero-shot accuracy on ImageNet (vs. CLIP's 72.7%) and excels in zero-shot retrieval, complex reasoning, and semantic segmentation. Additionally, SAIL improves the language-compatibility of vision encoders that in turn enhance the performance of multimodal large language models. The entire codebase and model weights are open-source: https://lezhang7.github.io/sail.github.io/

Figures (12)

• Figure 1: Conceptual Overview: Alignment probing evaluates the alignment potential of two pretrained uni-modal models.
• Figure 2: Linear alignment probing results trained with 2.2M paired data from CC3M. The radius represents the relative number of parameters in each model. The Y-axis indicates the zero-shot MSCOCO retrieval average R@10 performance. (Left) the X-axis shows kNN performance for various SSL models. (Right) the X-axis displays MTEB average scores across models.
• Figure 3: Winoground Results. CLIP(V) represents vision encoder and CLIP(T) represents text encoder from CLIP-L(WIT-400M). '+' indicates alignment probing with two models.
• Figure 4: SAIL Pipeline. Image-Text data is pre-encoded into embeddings. During alignment tuning, only embeddings and alignment layers are loaded to reduce GPU memory consumption and accelerate training speed.
• Figure 5: Results on standard retrieval, complex reasoning and visual-centric tasks. We report Recall@1 for MSCOCO and Flickr30k; Text, Image and Group scores for Winoground; and the average score for MMVP. $\dag$ ViT patch size is 14.