Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval

TL;DR

PrediCIR tackles zero-shot composed image retrieval by predicting target-content that is missing in the reference image, using a world-model-based latent predictor to produce a pseudo-word token $S_{*}$ for CLIP-based retrieval. The approach integrates World View Generation (creating source/target views from image-caption data) with an Image to Prediction-based Word Mapping (TCP+PMA) to fuse predicted and existing content before querying in a unified language space. It achieves state-of-the-art performance across six ZS-CIR tasks, including notable gains on FashionIQ, CIRR, CIRCO, and ImageNet domain conversion, while maintaining competitive inference speed. The work advances vision–language alignment by leveraging a latent world model for targeted content prediction, offering a pathway to more robust and scalable zero-shot retrieval with potential extension to multi-task and lightweight settings.

Abstract

Zero-Shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with a broad range of visual content manipulation intent across domain, scene, object, and attribute. The key challenge for ZS-CIR tasks is to modify a reference image according to manipulation text to accurately retrieve a target image, especially when the reference image is missing essential target content. In this paper, we propose a novel prediction-based mapping network, named PrediCIR, to adaptively predict the missing target visual content in reference images in the latent space before mapping for accurate ZS-CIR. Specifically, a world view generation module first constructs a source view by omitting certain visual content of a target view, coupled with an action that includes the manipulation intent derived from existing image-caption pairs. Then, a target content prediction module trains a world model as a predictor to adaptively predict the missing visual information guided by user intention in manipulating text at the latent space. The two modules map an image with the predicted relevant information to a pseudo-word token without extra supervision. Our model shows strong generalization ability on six ZS-CIR tasks. It obtains consistent and significant performance boosts ranging from 1.73% to 4.45% over the best methods and achieves new state-of-the-art results on ZS-CIR. Our code is available at https://github.com/Pter61/predicir.

Figures (10)

• Figure 1: Illustration of our motivation. (a) Prediction-free visual mapping. (b) Our prediction-based visual mapping. (c) ZS-CIR process results from different strategies.
• Figure 2: An overview of our PrediCIR model. Pre-training (left): Image to prediction-based word mapping aims to predict target-relevant missing visual content in latent space and map it with reference image content to a pseudo-word token $S_*$. Inference (right): Map the inference image to $S_*$ and form the composed query in a unified language space for ZS-CIR.
• Figure 3: Results of attribute manipulation on FashionIQ.
• Figure 4: Results of the object manipulation on CIRR.
• Figure 5: Retrieved results on the domain conversion task.