HyperspectralMAE: The Hyperspectral Imagery Classification Model using Fourier-Encoded Dual-Branch Masked Autoencoder
Wooyoung Jeong, Hyun Jae Park, Seonghun Jeong, Jong Wook Jang, Tae Hoon Lim, Dae Seoung Kim
TL;DR
HyperspectralMAE tackles the challenge of learning robust representations from high-dimensional hyperspectral data by extending masked autoencoder concepts to the spectral-spatial domain. It introduces a dual masking strategy (spatial and spectral), a Fourier-based spectral embedding to encode wavelength information, and a combined MSE+SAM loss to balance radiometric accuracy with spectral fidelity. The model scales to a foundation-size Transformer (~1.8×10^8 parameters) and is pre-trained on large unlabeled hyperspectral corpora (Hyperion and EnMAP), followed by fine-tuning on the Indian Pines dataset where it achieves state-of-the-art transfer performance. This approach demonstrates that dual-dimensional masking and wavelength-aware embeddings can significantly improve both reconstruction quality and downstream hyperspectral analysis, with practical impact for tasks like land-cover classification under limited labeled data.
Abstract
Hyperspectral imagery provides rich spectral detail but poses unique challenges because of its high dimensionality in both spatial and spectral domains. We propose \textit{HyperspectralMAE}, a Transformer-based foundation model for hyperspectral data that employs a \textit{dual masking} strategy: during pre-training we randomly occlude 50\% of spatial patches and 50\% of spectral bands. This forces the model to learn representations capable of reconstructing missing information across both dimensions. To encode spectral order, we introduce learnable harmonic Fourier positional embeddings based on wavelength. The reconstruction objective combines mean-squared error (MSE) with the spectral angle mapper (SAM) to balance pixel-level accuracy and spectral-shape fidelity. The resulting model contains about $1.8\times10^{8}$ parameters and produces 768-dimensional embeddings, giving it sufficient capacity for transfer learning. We pre-trained HyperspectralMAE on two large hyperspectral corpora -- NASA EO-1 Hyperion ($\sim$1\,600 scenes, $\sim$$3\times10^{11}$ pixel spectra) and DLR EnMAP Level-0 ($\sim$1\,300 scenes, $\sim$$3\times10^{11}$ pixel spectra) -- and fine-tuned it for land-cover classification on the Indian Pines benchmark. HyperspectralMAE achieves state-of-the-art transfer-learning accuracy on Indian Pines, confirming that masked dual-dimensional pre-training yields robust spectral-spatial representations. These results demonstrate that dual masking and wavelength-aware embeddings advance hyperspectral image reconstruction and downstream analysis.