Efficient Remote Sensing with Harmonized Transfer Learning and Modality Alignment

TL;DR

The paper tackles efficient multimodal transfer learning for remote sensing by addressing intra-modality embedding clustering that impedes cross-modal alignment. It introduces HarMA, a Harmonized Transfer Learning and Modality Alignment framework that uses a MultiModal Gated Adapter and a shared MMS-Adapter to enable cross-modal interaction with minimal parameter updates, guided by a unified objective combining $L_{ini}$, $L_{uniform}$, and a constraint $D(\theta,\theta^*)\le\delta$. The method combines $L_{\mathrm{ada-triplet}}$ and $L_{\mathrm{contrastive}}$ to achieve fine-grained alignment while preventing modality-wise over-clustering, achieving state-of-the-art retrieval on RSICD and RSITMD without external data and with substantially fewer tunable parameters than full fine-tuning. HarMA’s simplicity and compatibility suggest broad applicability to existing multimodal pretraining models, offering a practical route to deploy large models across diverse remote sensing tasks with reduced computational resources.

Abstract

With the rise of Visual and Language Pretraining (VLP), an increasing number of downstream tasks are adopting the paradigm of pretraining followed by fine-tuning. Although this paradigm has demonstrated potential in various multimodal downstream tasks, its implementation in the remote sensing domain encounters some obstacles. Specifically, the tendency for same-modality embeddings to cluster together impedes efficient transfer learning. To tackle this issue, we review the aim of multimodal transfer learning for downstream tasks from a unified perspective, and rethink the optimization process based on three distinct objectives. We propose "Harmonized Transfer Learning and Modality Alignment (HarMA)", a method that simultaneously satisfies task constraints, modality alignment, and single-modality uniform alignment, while minimizing training overhead through parameter-efficient fine-tuning. Remarkably, without the need for external data for training, HarMA achieves state-of-the-art performance in two popular multimodal retrieval tasks in the field of remote sensing. Our experiments reveal that HarMA achieves competitive and even superior performance to fully fine-tuned models with only minimal adjustable parameters. Due to its simplicity, HarMA can be integrated into almost all existing multimodal pretraining models. We hope this method can facilitate the efficient application of large models to a wide range of downstream tasks while significantly reducing the resource consumption. Code is available at https://github.com/seekerhuang/HarMA.

Figures (7)

• Figure 1: In remote sensing image-text retrieval, excessive clustering of the same modality sometimes leads to a decrease in performance. The experiment was conducted on the RSITMD dataset.
• Figure 2: The overall framework of the proposed method.
• Figure 3: The specific structure of the multimodal gated adapter.The overall structure is shown on the left, while the structure of the shared multimodal sub-adapter is displayed on the right.
• Figure 4: Image-to-Text Retrieval Visual Results. We compare the top-5 retrieved captions from HarMA (Ours) and the fully fine-tuned CLIP (Full-FT CLIP). Green text indicates correct retrieval, while red text indicates incorrect retrieval. The image on the far right corresponds to the text where retrieval errors occurred. Overall, HarMA demonstrates superior retrieval performance compared to Full-FT CLIP. Our method accurately captures the semantic elements, such as the tennis court in the first image, and associates them with the overall context. In contrast, Full-FT CLIP tends to overemphasize irrelevant details, mistakenly treating partial shadows and surrounding trees as the main subject matter.
• Figure 5: Text-to-Image Retrieval Visual Results. On the left side, we show the query text, and the ground-truth image is displayed below. The top-5 retrieved images based on the query text are presented on the right side, where green boxes indicate matches and red boxes indicate mismatches. For the mismatched retrieved images, we identify the main semantics of their associated text above the images. It is worth noting that in the rsitmd dataset, the relationship between images and texts is one-to-many, meaning that Image-to-Text can retrieve multiple results, while Text-to-Image has only one correct result.