Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval

TL;DR

This work tackles zero-shot composed image retrieval under practical deployment constraints by introducing asymmetric retrieval (light on the query side, heavy on the gallery side) and an adaptive token learner that maps an image to a sentence in a pre-trained vision-language embedding space. By concatenating the generated sentence tokens with a text modifier and leveraging a frozen, large VL foundation model, the approach learns from unlabeled images using global contrastive distillation (GCD) and local alignment regularization (LAR) to align the lightweight and VL representations. Experiments on CIRR, FashionIQ, and CIRCO show that ISA outperforms symmetric zero-shot baselines and prior ZSCIR methods, while offering improved efficiency and deployment flexibility on resource-constrained devices. The results demonstrate the practical impact of adaptive sentence-level representation and cross-modal alignment for robust, scalable CIR in real-world applications.

Abstract

The task of composed image retrieval (CIR) aims to retrieve images based on the query image and the text describing the users' intent. Existing methods have made great progress with the advanced large vision-language (VL) model in CIR task, however, they generally suffer from two main issues: lack of labeled triplets for model training and difficulty of deployment on resource-restricted environments when deploying the large vision-language model. To tackle the above problems, we propose Image2Sentence based Asymmetric zero-shot composed image retrieval (ISA), which takes advantage of the VL model and only relies on unlabeled images for composition learning. In the framework, we propose a new adaptive token learner that maps an image to a sentence in the word embedding space of VL model. The sentence adaptively captures discriminative visual information and is further integrated with the text modifier. An asymmetric structure is devised for flexible deployment, in which the lightweight model is adopted for the query side while the large VL model is deployed on the gallery side. The global contrastive distillation and the local alignment regularization are adopted for the alignment between the light model and the VL model for CIR task. Our experiments demonstrate that the proposed ISA could better cope with the real retrieval scenarios and further improve retrieval accuracy and efficiency.

Figures (8)

• Figure 1: Left: our training framework. Large visual encoder, large text encoder and matching encoder of pre-trained foundation model are frozen during training. The framework is trained with global contrastive distillation (GCD) and local alignment regularization (LAR). Right: adaptive token learner. Adaptive token learner learns to adaptively select salient visual patterns and map them to a sentence in the word embedding space of text encoder with attention mechanism.
• Figure 2: The inference workflow of our framework. The light encoder and the adaptive token learner are deployed on mobile device, to extract sentence tokens from the query image and concatenate them with the prompt and text modifier as the composed query. Pre-trained foundation model is deployed on the cloud server, and the large text encoder receives the uploaded composed query to extract the text feature for retrieval, and the large visual encoder is used to extract features of database images. The retrieval results are returned to the user side.
• Figure 3: The average and standard deviation of performance scores on three validation sets.
• Figure 4: Attention map of sentence tokens for different token lengths and retrieval results. Query image and text modifier are on the left side. Ground-truth target images are marked in green boxes.
• Figure 5: Some good retrieval examples and the attention map of sentence tokens on visual feature map and with modifier text. Ground-truth target images are marked in green boxes.