Neighbor-aware Instance Refining with Noisy Labels for Cross-Modal Retrieval

TL;DR

This work tackles cross-modal retrieval under noisy labels by introducing NIRNL, a unified framework that combines Cross-modal Margin Preserving (CMP) with Neighbor-aware Instance Refining (NIR). CMP tightens the relative margins between positive and negative cross-modal pairs, while NIR generates soft labels from global neighborhood consensus and partitions data into pure, hard, and noisy subsets for tailored optimization. The approach leverages semantic barycenters via a Wasserstein-based mechanism and applies subset-specific losses (CE for pure, weighted CE for hard, and MAE-based label correction for noisy) in combination with CMP, achieving state-of-the-art MAP on three benchmarks across varying noise rates. The results demonstrate improved robustness and data utilization, enabling more reliable cross-modal retrieval in realistic noisy-label settings.

Abstract

In recent years, Cross-Modal Retrieval (CMR) has made significant progress in the field of multi-modal analysis. However, since it is time-consuming and labor-intensive to collect large-scale and well-annotated data, the annotation of multi-modal data inevitably contains some noise. This will degrade the retrieval performance of the model. To tackle the problem, numerous robust CMR methods have been developed, including robust learning paradigms, label calibration strategies, and instance selection mechanisms. Unfortunately, they often fail to simultaneously satisfy model performance ceilings, calibration reliability, and data utilization rate. To overcome the limitations, we propose a novel robust cross-modal learning framework, namely Neighbor-aware Instance Refining with Noisy Labels (NIRNL). Specifically, we first propose Cross-modal Margin Preserving (CMP) to adjust the relative distance between positive and negative pairs, thereby enhancing the discrimination between sample pairs. Then, we propose Neighbor-aware Instance Refining (NIR) to identify pure subset, hard subset, and noisy subset through cross-modal neighborhood consensus. Afterward, we construct different tailored optimization strategies for this fine-grained partitioning, thereby maximizing the utilization of all available data while mitigating error propagation. Extensive experiments on three benchmark datasets demonstrate that NIRNL achieves state-of-the-art performance, exhibiting remarkable robustness, especially under high noise rates.

Figures (4)

• Figure 1: The framework of our proposed NIRNL. Our framework comprises two core modules operating in parallel: the Cross-modal Margin Preserving (CMP) module and the Neighbor-aware Instance Refining (NIR) module. The CMP module refines the global structure of the embedding space, promoting proximity between positive pairs (indicated by light yellow and light pink) while enforcing separation of negative pairs (indicated by dark yellow and dark pink). For clarity, only image samples are visualized in the NIR module. The NIR module initially computes the Wasserstein Barycenter of samples and generates soft labels through KNN. It subsequently partitions the dataset into pure, hard, and noisy subsets by evaluating the consistency between soft labels and ground-truth labels. Finally, we design three different loss functions for each subset to dig up as much semantic information as possible.
• Figure 2: Precision-recall curves under the 0.6 noise rate.
• Figure 3: The MAP scores versus epochs under the 0.6 noise rate.
• Figure 4: The number of instances versus epochs under the 0.6 noise rate.