PMPGuard: Catching Pseudo-Matched Pairs in Remote Sensing Image-Text Retrieval

TL;DR

<3-5 sentence high-level summary> PMPGuard tackles the prevalence of pseudo-matched pairs in remote-sensing image–text retrieval by introducing Cross-Gated Attention (CGA) to regulate cross-modal information flow and Positive–Negative Awareness Attention (PNAA) to explicitly mine reliable versus misleading cues. The framework combines an InfoNCE-based Inter-modal Aggregation Loss with a Positive–Negative Aware Loss, including a Gaussian-based mechanism to separate matched and mismatched fragments. Extensive experiments on RSICD, RSITMD, and RS5M demonstrate state-of-the-art robustness, especially as mismatch rates rise, validating the approach's ability to turn noisy supervision into informative signals. The results highlight PMPGuard’s practical impact for robust cross-modal RS understanding in real-world, imperfectly labeled datasets.

Abstract

Remote sensing (RS) image-text retrieval faces significant challenges in real-world datasets due to the presence of Pseudo-Matched Pairs (PMPs), semantically mismatched or weakly aligned image-text pairs, which hinder the learning of reliable cross-modal alignments. To address this issue, we propose a novel retrieval framework that leverages Cross-Modal Gated Attention and a Positive-Negative Awareness Attention mechanism to mitigate the impact of such noisy associations. The gated module dynamically regulates cross-modal information flow, while the awareness mechanism explicitly distinguishes informative (positive) cues from misleading (negative) ones during alignment learning. Extensive experiments on three benchmark RS datasets, i.e., RSICD, RSITMD, and RS5M, demonstrate that our method consistently achieves state-of-the-art performance, highlighting its robustness and effectiveness in handling real-world mismatches and PMPs in RS image-text retrieval tasks.

Figures (3)

• Figure 1: A simple example illustrates our key insight: Pseudo-Matched Pairs (PMPs), image–text samples with partial or incorrect alignment, are not merely noise but often contain latent semantic cues. Instead of discarding them, PMPGuard exploits these cues by rematching semantically relevant pairs (green links) and repelling irrelevant ones (red links), effectively transforming noisy supervision into useful alignment signals.
• Figure 2: PMPGuard overview: vision and text encoders extract features, then Cross-Gated Attention (CGA) suppresses mismatched cues and Positive–Negative Awareness Attention (PNAA) contrasts reliable/unreliable pairs, jointly optimized to mitigate pseudo-matched pairs and strengthen cross-modal alignment.
• Figure 3: PMPGuard rematches mismatched RS image-text pairs; green and red words denote correct and incorrect matches, respectively.