Imagine and Seek: Improving Composed Image Retrieval with an Imagined Proxy

TL;DR

This work tackles Zero-shot Composed Image Retrieval by addressing the gap between image and text alignment in existing methods. It introduces IP-CIR, a training-free framework that imagines a proxy image aligned with the query and relative captions, then builds robust proxy features by fusing proxy, query, and semantic perturbations, balanced with a novel similarity metric. Empirical results on CIRR, CIRCO, and FashionIQ demonstrate state-of-the-art gains and consistent improvements across backbones, validating the benefit of image-side imagined proxies for CIR. The approach highlights the practical potential of leveraging controllable image generation and LLM-driven layouts to enrich retrieval signals, while acknowledging time overhead and sensitivity to hyperparameters as areas for future refinement.

Abstract

The Zero-shot Composed Image Retrieval (ZSCIR) requires retrieving images that match the query image and the relative captions. Current methods focus on projecting the query image into the text feature space, subsequently combining them with features of query texts for retrieval. However, retrieving images only with the text features cannot guarantee detailed alignment due to the natural gap between images and text. In this paper, we introduce Imagined Proxy for CIR (IP-CIR), a training-free method that creates a proxy image aligned with the query image and text description, enhancing query representation in the retrieval process. We first leverage the large language model's generalization capability to generate an image layout, and then apply both the query text and image for conditional generation. The robust query features are enhanced by merging the proxy image, query image, and text semantic perturbation. Our newly proposed balancing metric integrates text-based and proxy retrieval similarities, allowing for more accurate retrieval of the target image while incorporating image-side information into the process. Experiments on three public datasets demonstrate that our method significantly improves retrieval performances. We achieve state-of-the-art (SOTA) results on the CIRR dataset with a Recall@K of 70.07 at K=10. Additionally, we achieved an improvement in Recall@10 on the FashionIQ dataset, rising from 45.11 to 45.74, and improved the baseline performance in CIRCO with a mAPK@10 score, increasing from 32.24 to 34.26.

Figures (13)

• Figure 1: Comparison of composed image retrieval between text-only retrieval and our methods. Traditional methods perform retrieval only in the text space, where CLIP text features may overlook some important semantic information. In contrast, our approach generates imagined proxy features, providing additional information that is often overlooked in text-only retrieval, thereby improving retrieval accuracy.
• Figure 1: Ablation results on the composition of robust proxy features in the CIRCO dataset. pi indicates proxy features, si represents semantic perturbation, and qi indicates the query features.
• Figure 2: Overview of our method. (a) represents our imagined proxy generation process. We use LLM to analyze the BLIP2-generated query image captions and the relative captions and infer the proxy layout. We then use the controllable generator to imagine the proxy images. (b) represents our process of constructing a robust proxy feature, and balancing the text and proxy similarities. We integrate proxy features, query image features as well as semantic perturbations into a robust proxy feature, and propose a balance metric for retrieval.
• Figure 2: Ablation result on the Robust Proxy. We present the visualization result of using different compositions (Qi represents only using query image, Si represents only semantic perturbation, and Pi represents only using the proxy image) of features in the robust proxy.
• Figure 3: Qualitative results of our method. We conducted experiments on the CIRCO validation dataset to observe in which cases our method improves retrieval results. In the blue section on the left, we display the query information used for retrieval, the ground truth target image, and our generated proxy image features. 'Query' represents the input query image, 'Caption' represents the relative text, and in the 'Proxy' section, we show two generated imagined proxies. In the red section on the right, the top and bottom rows display the top-5 retrieval results enhanced by our imagined proxies and the baseline's top-5 retrieval results, respectively.