SIGMAE: A Spectral-Index-Guided Foundation Model for Multispectral Remote Sensing

TL;DR

SIGMAE introduces Semantic Saliency-Guided Dynamic Token Masking (SSDTM), a curriculum-style strategy that quantifies each patch's semantic richness and internal heterogeneity to adaptively select the most informative tokens during training, and outperforms other pretrained geospatial foundation models.

Abstract

Pretraining and fine-tuning have emerged as a new paradigm in remote sensing image interpretation. Among them, Masked Autoencoder (MAE)-based pretraining stands out for its strong capability to learn general feature representations via reconstructing masked image regions. However, applying MAE to multispectral remote sensing images remains challenging due to complex backgrounds, indistinct targets, and the lack of semantic guidance during masking, which hinders the learning of underlying structures and meaningful spatial-spectral features. To address this, we propose a simple yet effective approach, Spectral Index-Guided MAE (SIGMAE), for multispectral image pretraining. The core idea is to incorporate domain-specific spectral indices as prior knowledge to guide dynamic token masking toward informative regions. SIGMAE introduces Semantic Saliency-Guided Dynamic Token Masking (SSDTM), a curriculum-style strategy that quantifies each patch's semantic richness and internal heterogeneity to adaptively select the most informative tokens during training. By prioritizing semantically salient regions and progressively increasing sample difficulty, SSDTM enhances spectrally rich and structurally aware representation learning, mitigates overfitting, and reduces redundant computation compared with random masking. Extensive experiments on five widely used datasets covering various downstream tasks, including scene classification, semantic segmentation, object extraction and change detection, demonstrate that SIGMAE outperforms other pretrained geospatial foundation models. Moreover, it exhibits strong spatial-spectral reconstruction capability, even with a 90% mask ratio, and improves complex target recognition under limited labeled data. The source codes and model weights will be released at https://github.com/zxk688/SIGMAE.

Figures (10)

• Figure 1: Performance comparison of remote sensing foundation models across five diverse datasets, where our SIGMAE achieved superior generalization capability.
• Figure 2: Overview of the proposed SIGMAE framework: an asymmetric encoder–decoder architecture equipped with Semantic Saliency-guided Dynamic Token Masking (SSDTM) that uses spectral-domain priors to adaptively select informative regions and enhance feature discriminability during reconstruction.
• Figure 3: Details of network structures of the (a) encoder, (b) reconstruction decoder and (c) for downstream tasks.
• Figure 4: Statistical characterization of spectral attributes and patch-level saliency indicators in BigEarthNet-S2 dataset. (a) Probability density functions of raw spectral indices (NDVI, NDWI, and NDBI) within the BigEarthNet-S2 dataset. (b) Distribution of the sum of absolute values derived from the indices, representing semantic richness. (c) Probability density of the resulting SSM.
• Figure 5: Spectral reconstruction performance comparison.