Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Perspective Subimage CLIP with Keyword Guidance for Remote Sensing Image-Text Retrieval

Yifan Li, Shiying Wang, Jianqiang Huang

TL;DR

The paper addresses the mismatch between coarse global alignment in RSITR and the dense, multi-scale semantics of remote sensing imagery, compounded by the high cost of full fine-tuning. It proposes MPS-CLIP, a parameter-efficient framework that uses an LLM to mine keywords guiding SamGeo to generate semantically relevant sub-perspectives, a Gated Global Attention (G^2A) adapter to adapt a frozen CLIP backbone, and a Multi-Perspective Representation (MPR) module to fuse local cues into robust embeddings, optimized with a hybrid objective that combines multi-perspective contrastive and weighted triplet losses. The approach delivers state-of-the-art results on RSICD and RSITMD, significantly outperforming full fine-tuning baselines and recent competitors while maintaining efficiency. These results demonstrate the value of keyword-guided multi-perspective modeling for fine-grained RSITR and suggest potential extensions to other dense prediction tasks in remote sensing.

Abstract

Vision-Language Pre-training (VLP) models like CLIP have significantly advanced Remote Sensing Image-Text Retrieval (RSITR). However, existing methods predominantly rely on coarse-grained global alignment, which often overlooks the dense, multi-scale semantics inherent in overhead imagery. Moreover, adapting these heavy models via full fine-tuning incurs prohibitive computational costs and risks catastrophic forgetting. To address these challenges, we propose MPS-CLIP, a parameter-efficient framework designed to shift the retrieval paradigm from global matching to keyword-guided fine-grained alignment. Specifically, we leverage a Large Language Model (LLM) to extract core semantic keywords, guiding the Segment Anything Model (SamGeo) to generate semantically relevant sub-perspectives. To efficiently adapt the frozen backbone, we introduce a Gated Global Attention (G^2A) adapter, which captures global context and long-range dependencies with minimal overhead. Furthermore, a Multi-Perspective Representation (MPR) module aggregates these local cues into robust multi-perspective embeddings. The framework is optimized via a hybrid objective combining multi-perspective contrastive and weighted triplet losses, which dynamically selects maximum-response perspectives to suppress noise and enforce precise semantic matching. Extensive experiments on the RSICD and RSITMD benchmarks demonstrate that MPS-CLIP achieves state-of-the-art performance with 35.18% and 48.40% mean Recall (mR), respectively, significantly outperforming full fine-tuning baselines and recent competitive methods. Code is available at https://github.com/Lcrucial1f/MPS-CLIP.

Multi-Perspective Subimage CLIP with Keyword Guidance for Remote Sensing Image-Text Retrieval

TL;DR

The paper addresses the mismatch between coarse global alignment in RSITR and the dense, multi-scale semantics of remote sensing imagery, compounded by the high cost of full fine-tuning. It proposes MPS-CLIP, a parameter-efficient framework that uses an LLM to mine keywords guiding SamGeo to generate semantically relevant sub-perspectives, a Gated Global Attention (G^2A) adapter to adapt a frozen CLIP backbone, and a Multi-Perspective Representation (MPR) module to fuse local cues into robust embeddings, optimized with a hybrid objective that combines multi-perspective contrastive and weighted triplet losses. The approach delivers state-of-the-art results on RSICD and RSITMD, significantly outperforming full fine-tuning baselines and recent competitors while maintaining efficiency. These results demonstrate the value of keyword-guided multi-perspective modeling for fine-grained RSITR and suggest potential extensions to other dense prediction tasks in remote sensing.

Abstract

Vision-Language Pre-training (VLP) models like CLIP have significantly advanced Remote Sensing Image-Text Retrieval (RSITR). However, existing methods predominantly rely on coarse-grained global alignment, which often overlooks the dense, multi-scale semantics inherent in overhead imagery. Moreover, adapting these heavy models via full fine-tuning incurs prohibitive computational costs and risks catastrophic forgetting. To address these challenges, we propose MPS-CLIP, a parameter-efficient framework designed to shift the retrieval paradigm from global matching to keyword-guided fine-grained alignment. Specifically, we leverage a Large Language Model (LLM) to extract core semantic keywords, guiding the Segment Anything Model (SamGeo) to generate semantically relevant sub-perspectives. To efficiently adapt the frozen backbone, we introduce a Gated Global Attention (G^2A) adapter, which captures global context and long-range dependencies with minimal overhead. Furthermore, a Multi-Perspective Representation (MPR) module aggregates these local cues into robust multi-perspective embeddings. The framework is optimized via a hybrid objective combining multi-perspective contrastive and weighted triplet losses, which dynamically selects maximum-response perspectives to suppress noise and enforce precise semantic matching. Extensive experiments on the RSICD and RSITMD benchmarks demonstrate that MPS-CLIP achieves state-of-the-art performance with 35.18% and 48.40% mean Recall (mR), respectively, significantly outperforming full fine-tuning baselines and recent competitive methods. Code is available at https://github.com/Lcrucial1f/MPS-CLIP.
Paper Structure (18 sections, 15 equations, 3 figures, 5 tables)

This paper contains 18 sections, 15 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Training-from-Scratch Methods
  4. Pre-training and Adaptation Methods
  5. Methods
  6. Overall Framework
  7. CLIP Encoder with the G$^2$A Adapter
  8. Multi-Perspective Representation (MPR) Module
  9. Multi Perspective Learning Objectives
  10. Experiments
  11. Datasets and Evaluation Metrics
  12. Implementation Details
  13. Comparison with SOTA Methods
  14. Ablation Studies and Discussion
  15. Ablation on the [CLS] Token in the CLIP
  16. ...and 3 more sections

Figures (3)

  • Figure 1: The overall framework of MPS-CLIP. The architecture integrates DeepSeek and SamGeo for semantic-aware patch generation.
  • Figure 2: Architecture of the proposed G$^2$A adapter.
  • Figure 3: Visual comparison of bidirectional retrieval results between "Ours" (MPS-CLIP) and HarMA. Green checks ($\checkmark$) and red crosses ($\times$) indicate semantically correct and incorrect retrieval results, respectively.