Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Redundancy-Aware Pretraining of Vision-Language Foundation Models in Remote Sensing

Mathis Jürgen Adler, Leonard Hackel, Gencer Sumbul, Begüm Demir

TL;DR

The paper tackles redundancy in remote-sensing vision-language pretraining caused by multiple captions per image. It introduces Weighted Feature Aggregation (WFA) with two weighting strategies: non-parametric uniqueness based on BLEU-4 and learning-based attention via Transformer encoders. The approach improves text-to-image retrieval performance while reducing pretraining and inference time compared to image replication, and provides guidelines for selecting weighting methods based on resource constraints. Public code is provided to enable scalable RS-VLM pretraining and future work aims to broaden downstream tasks and scale to larger data and architectures.

Abstract

The development of foundation models through pretraining of vision-language models (VLMs) has recently attracted great attention in remote sensing (RS). VLM pretraining aims to learn image and language alignments from a large number of image-text pairs. Each pretraining image is often associated with multiple captions containing redundant information due to repeated or semantically similar phrases, resulting in increased pretraining and inference time. To overcome this, we introduce a weighted feature aggregation (WFA) strategy for VLM pretraining in RS. Our strategy aims to extract and exploit complementary information from multiple captions per image while reducing redundancies through feature aggregation with importance weighting. To calculate adaptive importance weights for different captions of each image, we propose two techniques: (i) non-parametric uniqueness and (ii) learning-based attention. In the first technique, importance weights are calculated based on the bilingual evaluation understudy (BLEU) scores of the captions to emphasize unique sentences and reduce the influence of repetitive ones. In the second technique, importance weights are learned through an attention mechanism instead of relying on hand-crafted features. The effectiveness of the proposed WFA strategy with the two techniques is analyzed in terms of downstream performance on text-to-image retrieval in RS. Experimental results show that the proposed strategy enables efficient and effective pretraining of VLMs in RS. Based on the experimental analysis, we derive guidelines for selecting appropriate techniques depending on downstream task requirements and resource constraints. The code of this work is publicly available at https://git.tu-berlin.de/rsim/redundacy-aware-rs-vlm.

Redundancy-Aware Pretraining of Vision-Language Foundation Models in Remote Sensing

TL;DR

The paper tackles redundancy in remote-sensing vision-language pretraining caused by multiple captions per image. It introduces Weighted Feature Aggregation (WFA) with two weighting strategies: non-parametric uniqueness based on BLEU-4 and learning-based attention via Transformer encoders. The approach improves text-to-image retrieval performance while reducing pretraining and inference time compared to image replication, and provides guidelines for selecting weighting methods based on resource constraints. Public code is provided to enable scalable RS-VLM pretraining and future work aims to broaden downstream tasks and scale to larger data and architectures.

Abstract

The development of foundation models through pretraining of vision-language models (VLMs) has recently attracted great attention in remote sensing (RS). VLM pretraining aims to learn image and language alignments from a large number of image-text pairs. Each pretraining image is often associated with multiple captions containing redundant information due to repeated or semantically similar phrases, resulting in increased pretraining and inference time. To overcome this, we introduce a weighted feature aggregation (WFA) strategy for VLM pretraining in RS. Our strategy aims to extract and exploit complementary information from multiple captions per image while reducing redundancies through feature aggregation with importance weighting. To calculate adaptive importance weights for different captions of each image, we propose two techniques: (i) non-parametric uniqueness and (ii) learning-based attention. In the first technique, importance weights are calculated based on the bilingual evaluation understudy (BLEU) scores of the captions to emphasize unique sentences and reduce the influence of repetitive ones. In the second technique, importance weights are learned through an attention mechanism instead of relying on hand-crafted features. The effectiveness of the proposed WFA strategy with the two techniques is analyzed in terms of downstream performance on text-to-image retrieval in RS. Experimental results show that the proposed strategy enables efficient and effective pretraining of VLMs in RS. Based on the experimental analysis, we derive guidelines for selecting appropriate techniques depending on downstream task requirements and resource constraints. The code of this work is publicly available at https://git.tu-berlin.de/rsim/redundacy-aware-rs-vlm.
Paper Structure (6 sections, 7 equations, 2 figures, 1 table)

This paper contains 6 sections, 7 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Non-Parametric Uniqueness
  4. Learning-based Attention
  5. Experimental Results
  6. Conclusion

Figures (2)

  • Figure 1: An example image from the UCM-captions dataset qu2016deep with highly redundant (shown in red) and unique (shown in teal) information within its five captions.
  • Figure 2: An illustration of the proposed wfa strategy for pretraining the considered vlm (which consists of an image encoder $\phi_I$, a text encoder $\phi_\mathcal{C}$, and a projection head $\phi_p$ shared for image and text) with ntxent loss $\mathcal{L}_{\text{NTX}}$. The proposed WFA strategy exploits complementary information from all captions of an image at once, while reducing redundancies through feature aggregation with importance weighting.