Fine-Grained Zero-Shot Composed Image Retrieval with Complementary Visual-Semantic Integration

TL;DR

This work tackles zero-shot composed image retrieval by addressing the gap where prior methods rely solely on either visual tokens or modified captions, missing complementary information critical for fine-grained changes. It introduces CVSI, a three-component framework that separately extracts rich visual features and dense semantic descriptions from both query and database images, then fuses them via weighted complementary information retrieval in the CLIP space. The approach leverages a LinCIR-based mapping to pseudo tokens, BLIP-2 for dense captions, and an LLM to generate modified captions and likely-added objects, achieving new state-of-the-art results on CIRCO and CIRR, with strong performance on FashionIQ across multiple backbones and LLMs. The findings demonstrate that integrating complementary visual-semantic signals and object-level changes yields robust, cross-domain ZS-CIR capabilities, with practical implications for multimodal search and retrieval systems; code is publicly available.

Abstract

Zero-shot composed image retrieval (ZS-CIR) is a rapidly growing area with significant practical applications, allowing users to retrieve a target image by providing a reference image and a relative caption describing the desired modifications. Existing ZS-CIR methods often struggle to capture fine-grained changes and integrate visual and semantic information effectively. They primarily rely on either transforming the multimodal query into a single text using image-to-text models or employing large language models for target image description generation, approaches that often fail to capture complementary visual information and complete semantic context. To address these limitations, we propose a novel Fine-Grained Zero-Shot Composed Image Retrieval method with Complementary Visual-Semantic Integration (CVSI). Specifically, CVSI leverages three key components: (1) Visual Information Extraction, which not only extracts global image features but also uses a pre-trained mapping network to convert the image into a pseudo token, combining it with the modification text and the objects most likely to be added. (2) Semantic Information Extraction, which involves using a pre-trained captioning model to generate multiple captions for the reference image, followed by leveraging an LLM to generate the modified captions and the objects most likely to be added. (3) Complementary Information Retrieval, which integrates information extracted from both the query and database images to retrieve the target image, enabling the system to efficiently handle retrieval queries in a variety of situations. Extensive experiments on three public datasets (e.g., CIRR, CIRCO, and FashionIQ) demonstrate that CVSI significantly outperforms existing state-of-the-art methods. Our code is available at https://github.com/yyc6631/CVSI.

Figures (5)

• Figure 1: Comparison of methods for ZS-CIR. (a) Existing methods. (b) Our proposed method.
• Figure 2: Overview of the proposed CVSI. The CVSI consists of three key components: (1) Visual Information Extraction, (2) Semantic Information Extraction, and (3) Complementary Information Retrieval.
• Figure 3: The description of the designed prompt template $p_a$.
• Figure 4: Impact of different hyperparameters on model performance across CIRCO and FashionIQ. (a) Model performance on CIRCO with different hyperparameter combinations. (b) Model performance on FashionIQ with different hyperparameter combinations. (c) Model performance on CIRCO with different numbers of target captions.
• Figure 5: Qualitative analysis of top-5 retrieval results by our method and LDRE on the CIRCO test set, with ground-truth images highlighted in red.