Prompt-Guided Attention Head Selection for Focus-Oriented Image Retrieval

TL;DR

The paper tackles FOIR, where retrieving images containing a specific object from complex scenes requires user intent beyond global image features. It introduces Prompt-guided attention Head Selection (PHS), a training-free method that selects a subset of the last-layer ViT attention heads by matching their ROI-attention to visual prompts, thereby focusing on objects while preserving surrounding context. PHS operates in two modes (Query-Only and Query-DB) and does not alter input images or require fine-tuning, enabling practical deployment. Extensive experiments on COCO, PASCAL VOC, and Visual Genome across DINOv2 and CLIP backbones show consistent improvements in MP@100 and MAP@100, along with robustness to different prompts and prompt-noise, highlighting the method's potential for real-world, focus-aware image retrieval and better human-model attention alignment. The work presents a scalable, training-free enhancement to FOIR with broad applicability to multi-object, context-rich retrieval tasks.

Abstract

The goal of this paper is to enhance pretrained Vision Transformer (ViT) models for focus-oriented image retrieval with visual prompting. In real-world image retrieval scenarios, both query and database images often exhibit complexity, with multiple objects and intricate backgrounds. Users often want to retrieve images with specific object, which we define as the Focus-Oriented Image Retrieval (FOIR) task. While a standard image encoder can be employed to extract image features for similarity matching, it may not perform optimally in the multi-object-based FOIR task. This is because each image is represented by a single global feature vector. To overcome this, a prompt-based image retrieval solution is required. We propose an approach called Prompt-guided attention Head Selection (PHS) to leverage the head-wise potential of the multi-head attention mechanism in ViT in a promptable manner. PHS selects specific attention heads by matching their attention maps with user's visual prompts, such as a point, box, or segmentation. This empowers the model to focus on specific object of interest while preserving the surrounding visual context. Notably, PHS does not necessitate model re-training and avoids any image alteration. Experimental results show that PHS substantially improves performance on multiple datasets, offering a practical and training-free solution to enhance model performance in the FOIR task.

Figures (13)

• Figure 1: Overview of Focus-Oriented Image Retrieval (FOIR) task with illustration of visual prompting approach. FOIR task simulates real-world scenarios wherein (1) both query and retrieval database images often exhibit complexity, with multiple objects and intricate backgrounds; (2) users are visually interested in retrieving images containing specific object.
• Figure 2: Overview of Prompt-guided attention Head Selection (PHS). PHS performs the selection of transformer attention heads in the pretrained ViT model by matching their attention maps with user's visual prompt, which can take the form of a point, box, or segmentation. This empowers the model to focus on specific object of interest while preserving the surrounding visual context. Best viewed in color.
• Figure 3: Histogram of number of objects per query in datasets.
• Figure 4: Relative performance to CBIR (Model: CLIP large).
• Figure 5: Relative performance to CBIR (Model: DINOv2 large).