Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similarity

TL;DR

WeiMoCIR presents a training-free, zero-shot CIR approach that constructs a query by weighted fusion of a reference image and a text modifier and augments database representations with MLLM-generated captions. The method comprises two modules: Query Caption-Image Weighted Merge (QCI), which forms the query as \\mathbf{q} = (1-\\alpha) \\mathbf{v} + \\alpha \\mathbf{t}, and Ideal Caption-Image Distance Weighted Merge (ICD), which averages cosine similarities between the fused query and MLLM captions as well as images, culminating in the final score \\mathrm{s}_n = (1-\\beta) \\mathrm{s}^{Q2I}_n + \\beta \\mathrm{s}^{Q2C}_n. By combining query-to-image and query-to-caption similarities, WeiMoCIR achieves competitive results on FashionIQ and CIRR without any task-specific training, and benefits from larger VLM backbones and well-designed caption prompts. This training-free framework reduces computational demands and provides flexible applicability across datasets, with practical impact in rapid deployment of CIR systems. Future work will focus on enhancing the interpretation of nuanced multimodal queries and further exploiting prompt-driven caption diversity.

Abstract

Composed image retrieval (CIR), which formulates the query as a combination of a reference image and modified text, has emerged as a new form of image search due to its enhanced ability to capture user intent. However, training a CIR model in a supervised manner typically requires labor-intensive collection of (reference image, text modifier, target image) triplets. While existing zero-shot CIR (ZS-CIR) methods eliminate the need for training on specific downstream datasets, they still require additional pretraining on large-scale image datasets. In this paper, we introduce a training-free approach for ZS-CIR. Our approach, Weighted Modality fusion and similarity for CIR (WeiMoCIR), operates under the assumption that image and text modalities can be effectively combined using a simple weighted average. This allows the query representation to be constructed directly from the reference image and text modifier. To further enhance retrieval performance, we employ multimodal large language models (MLLMs) to generate image captions for the database images and incorporate these textual captions into the similarity computation by combining them with image information using a weighted average. Our approach is simple, easy to implement, and its effectiveness is validated through experiments on the FashionIQ and CIRR datasets. Code is available at https://github.com/whats2000/WeiMoCIR.

Figures (4)

• Figure 1: Overview of the proposed WeiMoCIR, a training-free approach for zero-shot composed image retrieval (ZS-CIR). Leveraging pretrained VLMs and MLLMs, our method comprises three modules: weighted modality fusion for query composition, enhanced representations through MLLM-generated image captions, and weighted modality similarity, which integrates both query-to-image and query-to-caption similarities for retrieval.
• Figure 2: Effects of $\alpha$ and $\beta$ on retrieval performance on FashionIQ. Left: average R@50 using CLIP ViT G/14. Right: average R@50 using BLIP ViT-B.
• Figure 3: Retrieval results of example queries from FashionIQ and CIRR. Our method with CLIP ViT L/14 successfully retrieves the desired images, highlighted with green boxes, demonstrating its ability to perform CIR for a wide variety of text modifiers.
• Figure 4: Failure cases of our method with CLIP ViT L/14 on example queries from FashionIQ and CIRR. Target images are highlighted with green boxes in the last column. As observed, the failures stem from difficulties in capturing more abstract concepts like 'simple print' and recognizing specific 'dog breeds'.