Feature Guided Masked Autoencoder for Self-supervised Learning in Remote Sensing

TL;DR

This work addresses the limitations of pixel-focused masked image modeling for remote sensing by introducing FG-MAE, which replaces raw pixel reconstruction targets with informative RS image features such as HOG and NDI for multispectral data and HOG for SAR. The FG-MAE framework preserves MAE’s efficient asymmetric encoder-decoder design while guiding representation learning through feature reconstruction, leading to improved downstream performance across MS and SAR tasks and datasets (BigEarthNet-MM, EuroSAT, DFC2020), with notable gains in SAR accuracy. The authors demonstrate scalability up to ViT-Huge (0.7B parameters) and release pretrained models along with the EuroSAT-SAR dataset to support future EO foundation-model research. They also discuss limitations, including underutilization of scale-invariant features like SIFT and challenges in fine-tuning large models, pointing to directions such as despeckling-based targets and more robust transfer strategies.

Abstract

Self-supervised learning guided by masked image modelling, such as Masked AutoEncoder (MAE), has attracted wide attention for pretraining vision transformers in remote sensing. However, MAE tends to excessively focus on pixel details, thereby limiting the model's capacity for semantic understanding, in particular for noisy SAR images. In this paper, we explore spectral and spatial remote sensing image features as improved MAE-reconstruction targets. We first conduct a study on reconstructing various image features, all performing comparably well or better than raw pixels. Based on such observations, we propose Feature Guided Masked Autoencoder (FG-MAE): reconstructing a combination of Histograms of Oriented Graidents (HOG) and Normalized Difference Indices (NDI) for multispectral images, and reconstructing HOG for SAR images. Experimental results on three downstream tasks illustrate the effectiveness of FG-MAE with a particular boost for SAR imagery. Furthermore, we demonstrate the well-inherited scalability of FG-MAE and release a first series of pretrained vision transformers for medium resolution SAR and multispectral images.

Figures (8)

• Figure 1: Sample data of the proposed FG-MAE method---columns from left to right: Sentinel-2 multispectral (MS) and Sentinel-1 (SAR) imagery, masked model inputs, model-reconstructed features (HOG: Histogram of Gradients, NDI: Normalized Difference Index). False color of the raw SAR image is coded by [VV, VH, (VV+VH)/2]. False color of the reconstructed MS NDI is coded by [NDVI, NDWI, NDBI].
• Figure 2: The general structure of the proposed FG-MAE method. We replace the reconstruction target of MAE he2022masked by remote sensing image features.
• Figure 3: Examples of FG-MAE reconstructed features. Every two rows represent one MS-SAR pair. From left to right, first row: MS image, MS HOG target, MS NDI target, SAR HOG target, SAR image; second row: MS image masked, MS HOG prediction, MS NDI reconstruction, SAR HOG prediction, SAR image masked.
• Figure 4: Examples of EuroSAT-SAR prediction results where FG-MAE gives the correct label while MAE doesn't. FG-MAE better captures semantics that are more distinguishable from the HOG features (e.g. a highway).
• Figure 5: Similar to MAE, FG-MAE scales well on BigEarthNet linear evaluation for both multispectral and SAR imagery.