Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

TL;DR

This paper addresses the mismatch between global image-text alignment and fine-grained visual concepts by showing that finer text descriptions align better with local image regions. It introduces Weighted Visual-Text Cross Alignment (WCA), which uses localized visual prompting to extract patches, then cross-aligns these patches with LLM-generated descriptions through a weighted similarity matrix and a weighted aggregation function. The key contributions include (i) a theoretical justification for focusing on local regions, (ii) a practical, training-free pipeline that computes a single cross-alignment score via weighted patch and text descriptions, and (iii) extensive experiments across multiple datasets and backbones demonstrating significant zero-shot improvements, competitive with few-shot methods. The approach offers a scalable, efficient path to improve vision-language models in real-world zero-shot scenarios without additional data or model training.

Abstract

It has recently been discovered that using a pre-trained vision-language model (VLM), e.g., CLIP, to align a whole query image with several finer text descriptions generated by a large language model can significantly enhance zero-shot performance. However, in this paper, we empirically find that the finer descriptions tend to align more effectively with local areas of the query image rather than the whole image, and then we theoretically validate this finding. Thus, we present a method called weighted visual-text cross alignment (WCA). This method begins with a localized visual prompting technique, designed to identify local visual areas within the query image. The local visual areas are then cross-aligned with the finer descriptions by creating a similarity matrix using the pre-trained VLM. To determine how well a query image aligns with each category, we develop a score function based on the weighted similarities in this matrix. Extensive experiments demonstrate that our method significantly improves zero-shot performance across various datasets, achieving results that are even comparable to few-shot learning methods.

Figures (14)

• Figure 1: Aligning an entire image with a detailed text description results in lower scaled cosine similarity, as shown on the left. Aligning the description with a specific image part, such as the detailed red patch (on the right), increases the score.
• Figure 2: We show different zero-shot visual-text alignment methods: (a) CLIP, (b) Visual-Text Alignment via LLMs menon2022visualpratt2023does, and (c) Weighted Visual-Text Cross Alignment (ours). Unlike (a) and (b), (c) utilizes a localized visual prompting technique to enhance alignment by ensuring that detailed descriptions match precisely with specific areas of the visual content.
• Figure 3: Overview of weighted visual-text cross alignment (WCA). The process begins with localized visual prompting, where the input image $\bm x$ is divided into localized patches, such as $\{\bm x_1, \bm x_2, \bm x_3\}$. These patches are encoded by an image encoder to produce visual features. The text prompting stage utilizes a large language model to generate detailed textual descriptions $\{\bm y_1, \bm y_2, \bm y_3\}$ for a given class label $\bm y$ (e.g., "woodpecker"). The WCA calculates alignment scores between visual features and textual features, using patch weights $\{w_1, w_2, w_3\}$ and text weights $\{v_1, v_2, v_3\}$. The final score is computed by summing the visual-text similarity matrix.
• Figure 4: Overview of text description weights for Jackfruit. This figure illustrates various text description weights based on their relevance to the visual cue "A photo of Jackfruit." The green lines represent the relative similarity scores, indicating how closely each description aligns with the visual cue. Longer green lines denote higher relevance, while shorter lines indicate lower relevance. Descriptions deemed irrelevant or incorrect are highlighted in red.
• Figure 5: We demonstrate the prediction and explanation of our methods and CLIP-Dmenon2022visual, in identifying and explaining a given image of a gas mask or respirator. The image is analyzed to predict its category, with the scaled cosine similarity scores between the image and various descriptions plotted for each method.

Theorems & Definitions (4)

• Theorem 4.1
• Definition 1.1
• Theorem 1.2
• proof