Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Satellite Tasks Need Special Pretraining?

Ani Vanyan, Alvard Barseghyan, Hakob Tamazyan, Tigran Galstyan, Vahan Huroyan, Naira Hovakimyan, Hrant Khachatrian

TL;DR

The paper investigates whether task-specific pretraining on remote-sensing data yields gains over general-purpose vision foundations for satellite imagery at small model scale. It introduces a resolution-generalization benchmark and compares iBOT variants trained on ImageNet versus MillionAID, with scale augmentation and an unlabeled change-detection head. Across classification and change-detection datasets, there is no consistent benefit from remote-sensing pretraining, suggesting that general-purpose foundation models remain competitive or superior at the ViT-B scale. The work highlights the limited impact of domain-specific pretraining for small models and emphasizes the value of augmentation and general-purpose models for satellite imagery tasks.

Abstract

Foundation models have advanced machine learning across various modalities, including images. Recently multiple teams trained foundation models specialized for remote sensing applications. This line of research is motivated by the distinct characteristics of remote sensing imagery, specific applications and types of robustness useful for satellite image analysis. In this work we systematically challenge the idea that specific foundation models are more useful than general-purpose vision foundation models, at least in the small scale. First, we design a simple benchmark that measures generalization of remote sensing models towards images with lower resolution for two downstream tasks. Second, we train iBOT, a self-supervised vision encoder, on MillionAID, an ImageNet-scale satellite imagery dataset, with several modifications specific to remote sensing. We show that none of those pretrained models bring consistent improvements upon general-purpose baselines at the ViT-B scale.

Do Satellite Tasks Need Special Pretraining?

TL;DR

The paper investigates whether task-specific pretraining on remote-sensing data yields gains over general-purpose vision foundations for satellite imagery at small model scale. It introduces a resolution-generalization benchmark and compares iBOT variants trained on ImageNet versus MillionAID, with scale augmentation and an unlabeled change-detection head. Across classification and change-detection datasets, there is no consistent benefit from remote-sensing pretraining, suggesting that general-purpose foundation models remain competitive or superior at the ViT-B scale. The work highlights the limited impact of domain-specific pretraining for small models and emphasizes the value of augmentation and general-purpose models for satellite imagery tasks.

Abstract

Foundation models have advanced machine learning across various modalities, including images. Recently multiple teams trained foundation models specialized for remote sensing applications. This line of research is motivated by the distinct characteristics of remote sensing imagery, specific applications and types of robustness useful for satellite image analysis. In this work we systematically challenge the idea that specific foundation models are more useful than general-purpose vision foundation models, at least in the small scale. First, we design a simple benchmark that measures generalization of remote sensing models towards images with lower resolution for two downstream tasks. Second, we train iBOT, a self-supervised vision encoder, on MillionAID, an ImageNet-scale satellite imagery dataset, with several modifications specific to remote sensing. We show that none of those pretrained models bring consistent improvements upon general-purpose baselines at the ViT-B scale.
Paper Structure (6 sections, 3 figures, 5 tables)

This paper contains 6 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Benchmark
  4. Pretraining iBOT on Remote Sensing Data
  5. Implementation Details
  6. Results

Figures (3)

  • Figure 1: The results of the baselines on our benchmark tasks for generalization across image resolution. The top row shows classification on RESISC and UC Merced, while the bottom row shows change detection on CDD and LEVIR-CD. X-axis: Scale of Distortions, Y-axis: Micro-F1 Scores.
  • Figure 2: iBOT pretraining architecture with an additional UperNet mask decoder that is trained using the "overlap loss". There are two global and eight local crops of the original image that pass through Teacher (T) and Student (S) networks. Dotted lines imply that only the representations of the last layers are used. Solid lines imply that representations of four layers are used (as an input to UperNet). Red lines correspond to patch representations, the blue lines correspond to CLS vectors.
  • Figure :