Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Scale Pretraining: Enhancing Self-Supervised Learning for Low-Resolution Satellite Imagery for Semantic Segmentation

John Waithaka, Gustave Bwirayesu, Moise Busogi

TL;DR

This work tackles the limited availability of pixel-wise annotations for remote-sensing semantic segmentation by introducing cross-scale pretraining that leverages real high-resolution (HR) and mid-resolution (MR) satellite imagery. A Spatial Affinity component, implemented as a student–teacher framework with a gram-based loss, transfers HR-rich spatial detail to MR representations, enabling improved MR downstream performance without losing MR spectral advantages. The method is validated across two SSL schemes, LatentMIM and I-JEPA, using the Sen2Venus HR/MR pairs and evaluated on Geo-Bench datasets, where SA-pretraining outperforms both HR-only and MR-only baselines. A key finding is that real HR data provide benefits beyond simple interpolation, and the study identifies ablations on downsampling and sampling strategies that influence the efficacy of cross-scale learning. Overall, the results demonstrate that effective cross-scale pretraining can enhance MR semantic segmentation and open avenues for reducing reliance on HR-MR image pairs in practice.

Abstract

Self-supervised pretraining in remote sensing is mostly done using mid-spatial resolution (MR) image datasets due to their high availability. Given the release of high-resolution (HR) datasets, we ask how HR datasets can be included in self-supervised pretraining to enhance MR image representation learning and downstream segmentation performance on MR tasks. We design a spatial affinity component that can be added to existing self-supervised learning frameworks and that uses HR imagery to learn better representations of MR imagery. We test the spatial affinity component on two self-supervised learning frameworks and show that it outperforms models pretrained on HR or MR images alone.

Cross-Scale Pretraining: Enhancing Self-Supervised Learning for Low-Resolution Satellite Imagery for Semantic Segmentation

TL;DR

This work tackles the limited availability of pixel-wise annotations for remote-sensing semantic segmentation by introducing cross-scale pretraining that leverages real high-resolution (HR) and mid-resolution (MR) satellite imagery. A Spatial Affinity component, implemented as a student–teacher framework with a gram-based loss, transfers HR-rich spatial detail to MR representations, enabling improved MR downstream performance without losing MR spectral advantages. The method is validated across two SSL schemes, LatentMIM and I-JEPA, using the Sen2Venus HR/MR pairs and evaluated on Geo-Bench datasets, where SA-pretraining outperforms both HR-only and MR-only baselines. A key finding is that real HR data provide benefits beyond simple interpolation, and the study identifies ablations on downsampling and sampling strategies that influence the efficacy of cross-scale learning. Overall, the results demonstrate that effective cross-scale pretraining can enhance MR semantic segmentation and open avenues for reducing reliance on HR-MR image pairs in practice.

Abstract

Self-supervised pretraining in remote sensing is mostly done using mid-spatial resolution (MR) image datasets due to their high availability. Given the release of high-resolution (HR) datasets, we ask how HR datasets can be included in self-supervised pretraining to enhance MR image representation learning and downstream segmentation performance on MR tasks. We design a spatial affinity component that can be added to existing self-supervised learning frameworks and that uses HR imagery to learn better representations of MR imagery. We test the spatial affinity component on two self-supervised learning frameworks and show that it outperforms models pretrained on HR or MR images alone.
Paper Structure (18 sections, 2 equations, 3 figures, 4 tables)

This paper contains 18 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-Supervised Learning for Remote Sensing
  4. Multi-resolution SSL for Remote Sensing
  5. Latent-Space Super-Resolution
  6. LatentMIM and I-JEPA
  7. Methodology
  8. Spatial Affinity Component
  9. Inputs
  10. Gram loss
  11. Implementation and Evaluation
  12. Results
  13. Super-Resolution
  14. Qualitative View
  15. Ablations
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Spatial affinity component samples patches from the high- and mid-resolution inputs. It uses the SSL framework's encoder to encode the lower resolution image and an added high-resolution teacher to encode the high-resolution input. The resulting representations from either encoder are used to compute the gram loss.
  • Figure 2: Patch sampling strategies used by the spatial affinity component and SSL frameworks
  • Figure 3: Unsupervised cluster maps of the patch representations of a Sentinel 2 image with $k=3$. Zoom in to see which model's representations are able to identify the distinct features circled in red.