NCL-CIR: Noise-aware Contrastive Learning for Composed Image Retrieval

TL;DR

This paper tackles Composed Image Retrieval (CIR) under realistic noisy supervision where multi-modal queries may be partially matched or mismatched to targets. It introduces Noise-aware Contrastive Learning for CIR (NCL-CIR), which combines a Weight Compensation Block (WCB) to refine multi-modal embeddings with a Noise-pair Filter Block (NFB) that uses a Gaussian Mixture Model to separate matched and noise pairs and generate soft labels. Training relies on a soft-label based Noise Contrastive Estimation loss ($L^{soft}_{NCE}$) that focuses on matched pairs, reducing the influence of noise. Empirically, NCL-CIR achieves strong performance gains on Fashion-IQ and Shoes, outperforming several baselines and demonstrating the value of explicit noise handling in CIR.

Abstract

Composed Image Retrieval (CIR) seeks to find a target image using a multi-modal query, which combines an image with modification text to pinpoint the target. While recent CIR methods have shown promise, they mainly focus on exploring relationships between the query pairs (image and text) through data augmentation or model design. These methods often assume perfect alignment between queries and target images, an idealized scenario rarely encountered in practice. In reality, pairs are often partially or completely mismatched due to issues like inaccurate modification texts, low-quality target images, and annotation errors. Ignoring these mismatches leads to numerous False Positive Pair (FFPs) denoted as noise pairs in the dataset, causing the model to overfit and ultimately reducing its performance. To address this problem, we propose the Noise-aware Contrastive Learning for CIR (NCL-CIR), comprising two key components: the Weight Compensation Block (WCB) and the Noise-pair Filter Block (NFB). The WCB coupled with diverse weight maps can ensure more stable token representations of multi-modal queries and target images. Meanwhile, the NFB, in conjunction with the Gaussian Mixture Model (GMM) predicts noise pairs by evaluating loss distributions, and generates soft labels correspondingly, allowing for the design of the soft-label based Noise Contrastive Estimation (NCE) loss function. Consequently, the overall architecture helps to mitigate the influence of mismatched and partially matched samples, with experimental results demonstrating that NCL-CIR achieves exceptional performance on the benchmark datasets.

Figures (2)

• Figure 1: (a). It presents the problem of partially matched or mismatched pairs existing in the previous methods. (b) A rough workflow of filtering the noise pairs and preserving the matched pairs in the proposed NFB in NCL-CIR.
• Figure 2: The workflow of NCL-CIR begins by encoding the modification text, reference image, and target image with CLIP to extract overall features and attention maps. Before the pairs are processed by the Noise-pair Filter Block (NFB), the feature embeddings pass through the Weight Compensation Block (WCB). This step produces refined embeddings, yielding multi-scale pair feature representations that enhance NFB's ability to filter noise pairs and preserve matched pairs. Additionally, NFB generates soft labels for the matched pairs, facilitating improved training within the soft-label based Noise Contrastive Estimation (NCE) loss function.