OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrieval

Abstract

Composed Image Retrieval (CIR) represents a novel retrieval paradigm that is capable of expressing users' intricate retrieval requirements flexibly. It enables the user to give a multimodal query, comprising a reference image and a modification text, and subsequently retrieve the target image. Notwithstanding the considerable advances made by prevailing methodologies, CIR remains in its nascent stages due to two limitations: 1) inhomogeneity between dominant and noisy portions in visual data is ignored, leading to query feature degradation, and 2) the priority of textual data in the image modification process is overlooked, which leads to a visual focus bias. To address these two limitations, this work presents a focus mapping-based feature extractor, which consists of two modules: dominant portion segmentation and dual focus mapping. It is designed to identify significant dominant portions in images and guide the extraction of visual and textual data features, thereby reducing the impact of noise interference. Subsequently, we propose a textually guided focus revision module, which can utilize the modification requirements implied in the text to perform adaptive focus revision on the reference image, thereby enhancing the perception of the modification focus on the composed features. The aforementioned modules collectively constitute the segmentatiOn-based Focus shiFt reviSion nETwork (\mbox{OFFSET}), and comprehensive experiments on four benchmark datasets substantiate the superiority of our proposed method. The codes and data are available on https://zivchen-ty.github.io/OFFSET.github.io/

Figures (4)

• Figure 1: (a) gives an example of the CIR task. (b) demonstrates the phenomenon of inhomogeneity in visual samples, where images frequently comprise dominant and noisy regions. (c) illustrates the advantages of applying text-priority during multimodal feature composition. The image caption treats "trees" as background noise information, which is inconsistent with the focus on modification text and may result in inaccurate composition results. However, when modification text is the primary objective, "trees" can be re-identified as the dominant region, thereby facilitating the construction of more accurate composed features.
• Figure 2: The proposed OFFSET consists of three key modules: (a) Dominant Portion Segmentation, (b) Dual Focus Mapping, and (c) Textually Guided Focus Revision, where (a) and (b) collectively form the feature extractor.
• Figure 3: Sensitivity to Focus Channel Number $P$ and the hyper-parameter $\mu$ on (a) FashionIQ, (b) Shoes, and (c) CIRR.
• Figure 4: Case study on (a) FashionIQ and (b) CIRR.