FOR: Finetuning for Object Level Open Vocabulary Image Retrieval

TL;DR

FOR tackles object-centric open-vocabulary image retrieval by fine-tuning a CLIP-based model on a target dataset while preserving vision-language alignment. It introduces SUM-CLIP, a decoder-enhanced head that summarizes image content into a small set of embeddings, and a multi-objective training regime that blends supervised base-label learning with pseudo-label guidance from Cluster-CLIP to support unseen concepts. The approach yields up to $8$ $mAP@50$ gains on novel categories across multiple datasets, reduces inference time by batching, and remains effective in semi-supervised settings with limited labeled data. This work offers a practical, scalable solution for open-vocabulary retrieval with compact embeddings and strong generalization to novel classes.

Abstract

As working with large datasets becomes standard, the task of accurately retrieving images containing objects of interest by an open set textual query gains practical importance. The current leading approach utilizes a pre-trained CLIP model without any adaptation to the target domain, balancing accuracy and efficiency through additional post-processing. In this work, we propose FOR: Finetuning for Object-centric Open-vocabulary Image Retrieval, which allows finetuning on a target dataset using closed-set labels while keeping the visual-language association crucial for open vocabulary retrieval. FOR is based on two design elements: a specialized decoder variant of the CLIP head customized for the intended task, and its coupling within a multi-objective training framework. Together, these design choices result in a significant increase in accuracy, showcasing improvements of up to 8 mAP@50 points over SoTA across three datasets. Additionally, we demonstrate that FOR is also effective in a semi-supervised setting, achieving impressive results even when only a small portion of the dataset is labeled.

Figures (9)

• Figure 1: Retrieval framework: Images are first encoded with a predefined number of embeddings and stored in a large-scale index. Subsequently, rapid and repeatable retrieval is performed by encoding text queries and conducting nearest neighbor searches. Notably, dual-encoder architectures enable separation into offline and online schemes, while scalability is enhanced by using a small number of embeddings per image.
• Figure 2: Comparison of designs: (a) Existing detection frameworks, either dense or RPN based, are impractical for retrieval due to their huge embedding representation; (b) Cluster-CLIP uses clustering for summarizing visual embeddings, but offers no finetunning capabilities; (c) SUM-CLIP employs learnable queries and enables gradient flow. Consequently, SUM-CLIP achieves higher accuracy and faster inference times.
• Figure 3: FOR overview. Our training framework combines a supervised loss using the dataset base categories labels, and a pseudo-labels loss leveraging ImageNet-21K classes. Pseudo labels are assigned by filtering ImageNet-21K classes based on the similarity between their textual embeddings and the image embeddings from Cluster-CLIP. On inference, FOR can be used with any textual query, while base and novel labels are used solely for evaluation purposes.
• Figure 4: SUM-CLIP head: CLIP (left) aims to represent the "average" semantics in images using $\bar{x}$ as a single query. Dense-CLIP (middle) focuses on local semantics induced by CLIP's original weights. SUM-CLIP (right) is designed to capture multiple objects by employing additional learnable queries and decoder layers preceding CLIP multi-head attention module.
• Figure 5: Qualitative Examples: Top-5 retrieved images from our overall system using COCO novel classes as queries. The index was created from 40K unlabeled COCO images with SUM-CLIP.