From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection

TL;DR

This paper tackles the challenge that random visual augmentations in vision-language pretraining can introduce background artifacts and degrade global semantic understanding. It proposes Attention-Based Selection (ABS), which uses DINO attention maps to guide cropping in both raw image space and intermediate feature space, and adds soft matching to filter text descriptions per crop. ABS is training-free and yields state-of-the-art zero-shot and out-of-distribution generalization across multiple CLIP backbones and datasets, often outperforming finetuning-based methods. The work provides extensive ablations and analyses showing the benefits of combining local detail with global context and validates the approach's robustness and transferability across VLM backbones.

Abstract

Pretrained vision-language models (VLMs), e.g., CLIP, demonstrate impressive zero-shot capabilities on downstream tasks. Prior research highlights the crucial role of visual augmentation techniques, like random cropping, in alignment with fine-grained class descriptions generated by large language models (LLMs), significantly enhancing zero-shot performance by incorporating multi-view information. However, the inherent randomness of these augmentations can inevitably introduce background artifacts and cause models to overly focus on local details, compromising global semantic understanding. To address these issues, we propose an \textbf{A}ttention-\textbf{B}ased \textbf{S}election (\textbf{ABS}) method from local details to global context, which applies attention-guided cropping in both raw images and feature space, supplement global semantic information through strategic feature selection. Additionally, we introduce a soft matching technique to effectively filter LLM descriptions for better alignment. \textbf{ABS} achieves state-of-the-art performance on out-of-distribution generalization and zero-shot classification tasks. Notably, \textbf{ABS} is training-free and even rivals few-shot and test-time adaptation methods. Our code is available at \href{https://github.com/BIT-DA/ABS}{\textcolor{darkgreen}{https://github.com/BIT-DA/ABS}}.

Figures (7)

• Figure 1: Random cropping for visual augmentation may capture background objects unrelated to the category ( red box), which have lower similarity to the text compared to semantically meaningful objects ( green box). Additionally, the randomness in crop size may result in background objects having a higher resolution than the main objects, leading to misjudgments when attempting to filter backgrounds based on image similarity.
• Figure 2: Similarity between the cropped image obtained in the image raw space and the cropped feature obtained at the feature map with the text descriptions. Although both crops can focus on the "eyes" as a local feature through cropping, the crop from the feature space retains the semantic information "bear", while the crop from the raw space misleads the model to identify it as "monkey".
• Figure 3: Framework overview. Raw space selection: We use DINO's attention map to guide image cropping, avoiding the inclusion of background objects. Feature selection: The original image is used as input and performs cropping on the feature map corresponding to the fine-grained selection before the final layer, to preserve global semantic information. Soft matching: We calculate a weight matrix to filter out irrelevant text descriptions for each crop, enabling better alignment.
• Figure 4: The sensitivity of three hyperparameter: crop ratio $\alpha$, number of crops $N$ and value of Top-$k$ on on ImageNet dataset using different CLIP backbones, ViT-B/16 and ViT-B/32, comparing with two baselines CLIP-D and CuPL.
• Figure 5: The visualization of the DINO and CLIP attention map and the cropped images guided by the attention maps.