iEBAKER: Improved Remote Sensing Image-Text Retrieval Framework via Eliminate Before Align and Keyword Explicit Reasoning

TL;DR

iEBAKER tackles the challenge of translating foundation models to remote-sensing image-text retrieval by filtering weakly correlated pairs before alignment and by injecting explicit keyword reasoning for fine-grained discrimination. The framework introduces Eliminate Before Align (EBA) with two schemes, Sort After Reversed Retrieval (SAR) as a post-processing step, and a Keyword Explicit Reasoning (KER) module, all trained with a noise-aware, EMA-regularized contrastive objective that includes an MLM-inspired keyword task. Empirical results on RSICD, RSITMD, and NWPU show state-of-the-art performance with substantially less training data (0.2M vs ~5M) and clear ablations validating each component. The approach eliminates the need for costly RS pretraining, delivering a data-efficient, high-precision RSITR solution with potential applicability to other cross-modal tasks.

Abstract

Recent studies focus on the Remote Sensing Image-Text Retrieval (RSITR), which aims at searching for the corresponding targets based on the given query. Among these efforts, the application of Foundation Models (FMs), such as CLIP, to the domain of remote sensing has yielded encouraging outcomes. However, existing FM based methodologies neglect the negative impact of weakly correlated sample pairs and fail to account for the key distinctions among remote sensing texts, leading to biased and superficial exploration of sample pairs. To address these challenges, we propose an approach named iEBAKER (an Improved Eliminate Before Align strategy with Keyword Explicit Reasoning framework) for RSITR. Specifically, we propose an innovative Eliminate Before Align (EBA) strategy to filter out the weakly correlated sample pairs, thereby mitigating their deviations from optimal embedding space during alignment.Further, two specific schemes are introduced from the perspective of whether local similarity and global similarity affect each other. On this basis, we introduce an alternative Sort After Reversed Retrieval (SAR) strategy, aims at optimizing the similarity matrix via reverse retrieval. Additionally, we incorporate a Keyword Explicit Reasoning (KER) module to facilitate the beneficial impact of subtle key concept distinctions. Without bells and whistles, our approach enables a direct transition from FM to RSITR task, eliminating the need for additional pretraining on remote sensing data. Extensive experiments conducted on three popular benchmark datasets demonstrate that our proposed iEBAKER method surpasses the state-of-the-art models while requiring less training data. Our source code will be released at https://github.com/zhangy0822/iEBAKER.

Figures (7)

• Figure 1: Illustration of weakly correlated pairs. The left "viaduct" image is captioned as "A large number of trees are planted on both sides of the road", whereas they are weakly correlated. In contrast, the right "a medium residential" image is strongly correlated with above caption but considered as a negative image-text pair.
• Figure 2: Comparison between our EBAKER ji2024eliminate and iEBAKER with the existing FM based methods on RSITR task. (a) A significant volume of model-annotated remote sensing image-text pairs are employed to adapt the Foundation Model (FM) into a Remote Sensing Foundation Model (RSFM). Subsequently, the RSFM undergoes a further transformation into a Remote Sensing Image-Text Retrieval Model (RSITRM) through additional coarse-grained contrastive learning on the RSITR dataset. (b) In contrast, our approach achieves a direct one-step transition from FM to RSITRM by integrating the Eliminate Before Align (EBA) strategy and the Keyword Explicit Reasoning (KER) module, streamlining the process and enhancing retrieval accuracy.
• Figure 3: Overview of our iEBAKER approach, which is composed of four key components: A. Feature Extractor: CLIP radford2021learning is employed as the encoder for both visual and textual modalities. We also conduct word frequency analysis to mask critical keywords. This process yields visual features, textual features, and masked textual features. B. Eliminate Before Align: Prior to the alignment step, we eliminate positive sample pairs that exhibit low global similarity, aiming at mitigating the negative impact of the weakly correlated pairs. This improved version introduce two specific schemes from the perspective of whether local similarity and global similarity affect each other, i.e., the EBA-Joint and the EBA-Split. C. Sort After Reversed Retrieval: A novel post-processing strategy is applied to optimize local and global similarities, respectively. D. Keyword Explicit Reasoning: To capture subtle distinctions among remote sensing images, we implement a keyword prediction technique that highlights key concepts, promoting more accurate and fine-grained contrastive learning.
• Figure 4: Explaination of Eliminate Before Align strategy. 1) Establish similarity bank to store all global or local similarities on-the-fly. 2) Sort all the similarities. 3) Set the threshold based on the drop ratio. 4) Eliminate rows within a batch corresponding to Image-to-Text or Text-to-Image pairs with similarities below the threshold before the alignment in the next epoch.
• Figure 5: Impacts of (a) Drop epoch and (b) Drop ratio. Note that R@k (k=1,5,10) refer to the left vertical coordinates while meanR refers to the right vertical coordinates.