Denoise-I2W: Mapping Images to Denoising Words for Accurate Zero-Shot Composed Image Retrieval

TL;DR

This paper proposes a novel denoising image-to-word mapping approach, named Denoise-I2W, for mapping images into denoising pseudo-word tokens that, without intention-irrelevant visual information, enhance accurate ZS-CIR.

Abstract

Zero-Shot Composed Image Retrieval (ZS-CIR) supports diverse tasks with a broad range of visual content manipulation intentions that can be related to domain, scene, object, and attribute. A key challenge for ZS-CIR is to accurately map image representation to a pseudo-word token that captures the manipulation intention relevant image information for generalized CIR. However, existing methods between the retrieval and pre-training stages lead to significant redundancy in the pseudo-word tokens. In this paper, we propose a novel denoising image-to-word mapping approach, named Denoise-I2W, for mapping images into denoising pseudo-word tokens that, without intention-irrelevant visual information, enhance accurate ZS-CIR. Specifically, a pseudo triplet construction module first automatically constructs pseudo triples (\textit{i.e.,} a pseudo-reference image, a pseudo-manipulation text, and a target image) for pre-training the denoising mapping network. Then, a pseudo-composed mapping module maps the pseudo-reference image to a pseudo-word token and combines it with the pseudo-manipulation text with manipulation intention. This combination aligns with the target image, facilitating denoising intention-irrelevant visual information for mapping. Our proposed Denoise-I2W is a model-agnostic and annotation-free approach. It demonstrates strong generalization capabilities across three state-of-the-art ZS-CIR models on four benchmark datasets. By integrating Denoise-I2W with existing best models, we obtain consistent and significant performance boosts ranging from 1.45\% to 4.17\% over the best methods without increasing inference costs. and achieve new state-of-the-art results on ZS-CIR. Our code is available at \url{https://github.com/Pter61/denoise-i2w-tmm}.

Figures (9)

• Figure 1: An illustration of our motivation. (a) Noising context-consistent visual mapping. (b) Our denoising context-complementary visual mapping. (c) ZS-CIR process with the results of different mapping strategies.
• Figure 2: An overview of our Denoise-I2W. Pseudo Triplet Construction (left): Automatically constructing pseudo triples with partial-cropped images and style-transfer images from existing image-caption pairs. Pre-training and Inference (right): Mapping the intention-based denoising visual content to a pseudo-word token $S_*$ and forming the composed query in a unified language space for ZS-CIR.
• Figure 3: Visualization of retrieved results on the object manipulation task. Our Denosing-I2W enables ZS-CIR models to map intent-specific object/scene information, thereby enhancing the model’s ability to understand complex objects/scenes.
• Figure 4: Visualization of retrieved results of attribute manipulation.
• Figure 5: Retrieved results on the object composition task.