Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models

TL;DR

BiFTA tackles redundancy in fine-grained text-visual alignment for CLIP-based vision-language models by introducing two refinements: View Refinement (IoU-based deduplication of cropped image patches) and Description Refinement (cosine-similarity pruning of diverse, LLM-generated descriptions). This dual refinement yields deduplicated, semantically diverse visual views and textual descriptions, enabling more accurate cross-modal alignment and improved zero-shot classification across six benchmarks and multiple CLIP backbones. The approach demonstrates consistent gains over state-of-the-art cross-alignment methods with modest offline preprocessing costs and no inference-time overhead, highlighting its practical impact and broad applicability to prompt-learning frameworks. Overall, BiFTA provides a principled, efficient solution to redundancy in visual-text alignment, enabling more reliable fine-grained recognition in real-world settings.

Abstract

Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-language models (e.g., CLIP). However, we find that both fine-grained text descriptions and localized image patches often contain redundant information, making text-visual alignment less effective. In this paper, we tackle this issue from two perspectives: \emph{View Refinement} and \emph{Description refinement}, termed as \textit{\textbf{Bi}-refinement for \textbf{F}ine-grained \textbf{T}ext-visual \textbf{A}lignment} (BiFTA). \emph{View refinement} removes redundant image patches with high \emph{Intersection over Union} (IoU) ratios, resulting in more distinctive visual samples. \emph{Description refinement} removes redundant text descriptions with high pairwise cosine similarity, ensuring greater diversity in the remaining descriptions. BiFTA achieves superior zero-shot performance on 6 benchmark datasets for both ViT-based and ResNet-based CLIP, justifying the necessity to remove redundant information in visual-text alignment.

Figures (9)

• Figure 1: Weaknesses of weighted visual-text cross alignment li2024visual. Weakness 1: Pairwise similarity scores of highly overlapping crop bounding boxes. We demonstrate that image patches A, B, and C, exhibiting significant overlap and redundancy, which provide limited semantic information and consequently contribute minimally to accurate classification. Weakness 2: Redundant textual descriptions generated by LLM. We gather textual descriptions from previous work and demonstrate that a significant portion of these descriptions are redundant for a given category, thereby diluting the contribution of meaningful and informative descriptions.
• Figure 2: We select an ImageNet image of a goose and display the Top-10 predictions ranked by WCA scores. The scores are normalized using softmax and its distribution is visualized using color intensity. The results show that applying an IoU-based filter to eliminate duplicated image patches significantly enhances the precision of WCA scoring.
• Figure 3: An Overview of BiFTA. To reduce potential redundancy in views and descriptions, the randomly cropped views undergo filtering with the IoU filter (Section \ref{['Sec: 4.1']}), while the randomly sampled description texts are processed using the CoS filter (Section \ref{['Sec: 4.2']}) when computing the similarity between a single image and a single label. The similarity score is then calculated on the refined views and descriptions.
• Figure 4: A visualization comparing the effectiveness of VR and DR in BiFTA against WCA. Left: with an IoU filter, the cropped samples exhibit diverse and distinctive localized features. Right: with a CoS filter, the texts can describe various local features of a category.
• Figure 5: Left: Accuracy of using alternative IoU thresholds on DTD dataset with CLIP (B/32). Right: Accuracy of changing the size of $|V|$ on DTD dataset with CLIP (B/32).

Theorems & Definitions (2)

• Definition 1
• Definition 2