Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval
Siting Li, Xiang Gao, Simon Shaolei Du
TL;DR
Attribute-focused image retrieval aims to match images to queries that emphasize specific visual attributes rather than global semantics. The authors introduce COCO-Facet to benchmark this setting and demonstrate that both CLIP-like and current MLLM-based retrievers struggle with fine-grained attributes. They propose promptable image embeddings that condition on GPT-generated prompts to highlight target attributes, showing improved recall across diverse attribute types and image pools. To enable practical deployment, they propose pre-processing prompts and a test-time linear approximation, achieving significant recall gains with manageable compute. Overall, the work advances fine-grained, attribute-grounded T2I retrieval with tangible efficiency strategies and provides a benchmark and codebase for future research.
Abstract
While an image is worth more than a thousand words, only a few provide crucial information for a given task and thus should be focused on. In light of this, ideal text-to-image (T2I) retrievers should prioritize specific visual attributes relevant to queries. To evaluate current retrievers on handling attribute-focused queries, we build COCO-Facet, a COCO-based benchmark with 9,112 queries about diverse attributes of interest. We find that CLIP-like retrievers, which are widely adopted due to their efficiency and zero-shot ability, have poor and imbalanced performance, possibly because their image embeddings focus on global semantics and subjects while leaving out other details. Notably, we reveal that even recent Multimodal Large Language Model (MLLM)-based, stronger retrievers with a larger output dimension struggle with this limitation. Hence, we hypothesize that retrieving with general image embeddings is suboptimal for performing such queries. As a solution, we propose to use promptable image embeddings enabled by these multimodal retrievers, which boost performance by highlighting required attributes. Our pipeline for deriving such embeddings generalizes across query types, image pools, and base retriever architectures. To enhance real-world applicability, we offer two acceleration strategies: Pre-processing promptable embeddings and using linear approximations. We show that the former yields a 15% improvement in Recall@5 when prompts are predefined, while the latter achieves an 8% improvement when prompts are only available during inference.