Dual-Domain Masked Image Modeling: A Self-Supervised Pretraining Strategy Using Spatial and Frequency Domain Masking for Hyperspectral Data

TL;DR

This work tackles the limited labeled data problem in hyperspectral image classification by introducing Spatial-Frequency Masked Image Modeling (SFMIM), a self-supervised pretraining strategy for transformer-based encoders. SFMIM masks data in two domains: spatial patches and spectral frequencies, using a Fourier-domain masking scheme with cutoff $\alpha=\gamma \lceil B/2\rceil$ and a patch-level spatial mask, then trains a single ViT-like encoder with a lightweight decoder under a mean squared error loss. The approach yields state-of-the-art results on three public benchmarks (Indian Pines, University of Pavia, Houston 2013) and shows faster convergence during fine-tuning compared to prior methods like MAEST and FactoFormer. By leveraging abundant unlabeled HSIs, SFMIM reduces annotation requirements while delivering efficient, robust spatial-spectral representations for downstream classification.

Abstract

Hyperspectral images (HSIs) capture rich spectral signatures that reveal vital material properties, offering broad applicability across various domains. However, the scarcity of labeled HSI data limits the full potential of deep learning, especially for transformer-based architectures that require large-scale training. To address this constraint, we propose Spatial-Frequency Masked Image Modeling (SFMIM), a self-supervised pretraining strategy for hyperspectral data that utilizes the large portion of unlabeled data. Our method introduces a novel dual-domain masking mechanism that operates in both spatial and frequency domains. The input HSI cube is initially divided into non-overlapping patches along the spatial dimension, with each patch comprising the entire spectrum of its corresponding spatial location. In spatial masking, we randomly mask selected patches and train the model to reconstruct the masked inputs using the visible patches. Concurrently, in frequency masking, we remove portions of the frequency components of the input spectra and predict the missing frequencies. By learning to reconstruct these masked components, the transformer-based encoder captures higher-order spectral-spatial correlations. We evaluate our approach on three publicly available HSI classification benchmarks and demonstrate that it achieves state-of-the-art performance. Notably, our model shows rapid convergence during fine-tuning, highlighting the efficiency of our pretraining strategy.

Figures (1)

• Figure 1: An overview of the proposed SFMIM network architecture. The input hyperspectral cube is split into non-overlapping patches along the spatial dimension, masked in both spatial and frequency domains, and processed by a transformer encoder. A lightweight decoder reconstructs the masked input with MSE used as the reconstruction loss.