Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval

TL;DR

This work tackles partially mismatched pairs (PMPs) in cross-modal retrieval by introducing L2RM, an OT-based framework that rematches mismatched image-text pairs. It jointly learns a self-supervised cost function for optimal transport and employs a partial OT formulation with a masking constraint to refine alignments while limiting false positives. The approach demonstrates robust improvements over state-of-the-art baselines across Flickr30K, MS-COCO, and CC152K, with ablations confirming the importance of the learnable cost and partial transport components. By effectively leveraging PMPs, L2RM enables more data-efficient cross-modal learning and improved retrieval performance in web-sourced datasets.

Abstract

Collecting well-matched multimedia datasets is crucial for training cross-modal retrieval models. However, in real-world scenarios, massive multimodal data are harvested from the Internet, which inevitably contains Partially Mismatched Pairs (PMPs). Undoubtedly, such semantical irrelevant data will remarkably harm the cross-modal retrieval performance. Previous efforts tend to mitigate this problem by estimating a soft correspondence to down-weight the contribution of PMPs. In this paper, we aim to address this challenge from a new perspective: the potential semantic similarity among unpaired samples makes it possible to excavate useful knowledge from mismatched pairs. To achieve this, we propose L2RM, a general framework based on Optimal Transport (OT) that learns to rematch mismatched pairs. In detail, L2RM aims to generate refined alignments by seeking a minimal-cost transport plan across different modalities. To formalize the rematching idea in OT, first, we propose a self-supervised cost function that automatically learns from explicit similarity-cost mapping relation. Second, we present to model a partial OT problem while restricting the transport among false positives to further boost refined alignments. Extensive experiments on three benchmarks demonstrate our L2RM significantly improves the robustness against PMPs for existing models. The code is available at https://github.com/hhc1997/L2RM.

Figures (7)

• Figure 1: A toy example to illustrate our idea. The potential semantic similarity among unpaired samples makes it possible to excavate useful knowledge from mismatched pairs. Our L2RM aims to rematch PMPs by generating a refined alignment that brings relevant cross-modal samples (green links) together while repelling irrelevant ones (red links) away from each other. We also show some real-world rematched cases for our L2RM in \ref{['fig:case_study']}.
• Figure 2: Overview of the learnable cost function with self-supervised learning. The up part illustrates the reconstructed pairs that only $(V_4,T_1)$, $(V_2,T_3)$, and $(V_{N_b},T_{N_b})$ are the reserved matching ones. Then, the matching matrix is viewed as supervision to guide the cost function from the explicit similarity-cost mapping relation through an OT loss (the down part).
• Figure 3: Illustration of the proposed rematching loss. For the mismatched pairs, we formalize a partial OT problem with positive pairs masked to generate the refined alignment in each batch. The refined alignment provides a more reliable matching relation to supervise the mismatched pairs. Then, we compute the symmetric KL-divergence to optimize the retrieval model $(f_v,f_t,g)$.
• Figure 4: Parameter analysis of L2RM-SGR in terms of recall scores on the testing set of Flickr30K under 0.2 MRate.
• Figure 5: The ability of our L2RM to rematch the mismatched visual-text samples. The figure shows some representative rematched pairs for L2RM-SGR on the training set of CC152K dataset. We highlight the matched words in green and the mismatched words in red.