Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Subimage Overlap Prediction: Task-Aligned Self-Supervised Pretraining For Semantic Segmentation In Remote Sensing Imagery

Lakshay Sharma, Alex Marin

TL;DR

The paper addresses label-efficient semantic segmentation in remote sensing by introducing Subimage Overlap Prediction, a task-aware self-supervised pretraining objective that localizes a subimage within its parent image to learn transferable spatial features. Implemented with two architectures (DINOv2 ViT-S/14 and a dual-encoder ResNet-50), the method pretrains on a small RS dataset (LandCoverAI) and yields faster downstream convergence and competitive IoU on land-cover segmentation, including cross-dataset transfer to LoveDA and DeepGlobe. Compared to ImageNet and larger SSL baselines, Subimage Overlap achieves comparable or better downstream performance with substantially less pretraining data, and shows particular advantage when labeled data are scarce. This work demonstrates that task-aligned self-supervision can markedly improve data efficiency for dense RS prediction tasks and points to broader applicability to other RS domains and tasks.

Abstract

Self-supervised learning (SSL) methods have become a dominant paradigm for creating general purpose models whose capabilities can be transferred to downstream supervised learning tasks. However, most such methods rely on vast amounts of pretraining data. This work introduces Subimage Overlap Prediction, a novel self-supervised pretraining task to aid semantic segmentation in remote sensing imagery that uses significantly lesser pretraining imagery. Given an image, a sub-image is extracted and the model is trained to produce a semantic mask of the location of the extracted sub-image within the original image. We demonstrate that pretraining with this task results in significantly faster convergence, and equal or better performance (measured via mIoU) on downstream segmentation. This gap in convergence and performance widens when labeled training data is reduced. We show this across multiple architecture types, and with multiple downstream datasets. We also show that our method matches or exceeds performance while requiring significantly lesser pretraining data relative to other SSL methods. Code and model weights are provided at \href{https://github.com/sharmalakshay93/subimage-overlap-prediction}{github.com/sharmalakshay93/subimage-overlap-prediction}.

Subimage Overlap Prediction: Task-Aligned Self-Supervised Pretraining For Semantic Segmentation In Remote Sensing Imagery

TL;DR

The paper addresses label-efficient semantic segmentation in remote sensing by introducing Subimage Overlap Prediction, a task-aware self-supervised pretraining objective that localizes a subimage within its parent image to learn transferable spatial features. Implemented with two architectures (DINOv2 ViT-S/14 and a dual-encoder ResNet-50), the method pretrains on a small RS dataset (LandCoverAI) and yields faster downstream convergence and competitive IoU on land-cover segmentation, including cross-dataset transfer to LoveDA and DeepGlobe. Compared to ImageNet and larger SSL baselines, Subimage Overlap achieves comparable or better downstream performance with substantially less pretraining data, and shows particular advantage when labeled data are scarce. This work demonstrates that task-aligned self-supervision can markedly improve data efficiency for dense RS prediction tasks and points to broader applicability to other RS domains and tasks.

Abstract

Self-supervised learning (SSL) methods have become a dominant paradigm for creating general purpose models whose capabilities can be transferred to downstream supervised learning tasks. However, most such methods rely on vast amounts of pretraining data. This work introduces Subimage Overlap Prediction, a novel self-supervised pretraining task to aid semantic segmentation in remote sensing imagery that uses significantly lesser pretraining imagery. Given an image, a sub-image is extracted and the model is trained to produce a semantic mask of the location of the extracted sub-image within the original image. We demonstrate that pretraining with this task results in significantly faster convergence, and equal or better performance (measured via mIoU) on downstream segmentation. This gap in convergence and performance widens when labeled training data is reduced. We show this across multiple architecture types, and with multiple downstream datasets. We also show that our method matches or exceeds performance while requiring significantly lesser pretraining data relative to other SSL methods. Code and model weights are provided at \href{https://github.com/sharmalakshay93/subimage-overlap-prediction}{github.com/sharmalakshay93/subimage-overlap-prediction}.
Paper Structure (14 sections, 2 equations, 5 figures, 5 tables)

This paper contains 14 sections, 2 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Subimage Overlap Pretraining
  4. Architecture
  5. DinoV2 backbone
  6. ResNet-50 Backbone
  7. Training Subimage Overlap Prediction
  8. Evaluation / Results - Subimage Overlap Prediction
  9. Downstream transfer learning
  10. Ablating the Task-Aware Pretraining for Land-Cover Segmentation
  11. Evaluation / results
  12. Varying Downstream Data
  13. External SSL Comparisons
  14. Conclusion

Figures (5)

  • Figure 1: Overview of the Subimage Overlap pretraining process. The model receives an input image and a selected subimage, and learns to predict a binary mask indicating the subimage’s location within the image.
  • Figure 2: Subimage Overlap prediction examples. The green square represents the selected subimage / ground truth; the red mask shows the predictions by a DinoV2 backbone model.
  • Figure 3: Subimage Overlap pretraining with LandCoverAI: train-val mIoU curves and predictions by epoch. Green boxes represent the selected subimage / ground truth; red mask represents the prediction after $k$ training epochs.
  • Figure 4: LandCoverAI segmentation: IoU convergence and best performance with varying amounts of labeled training samples.
  • Figure 5: LoveDA and DeepGlobe segmentation: Convergence of validation IoU over training epochs. Pretrained model uses LandCoverAI for pretraining.