Cross-Domain Transfer with Self-Supervised Spectral-Spatial Modeling for Hyperspectral Image Classification
Jianshu Chao, Tianhua Lv, Qiqiong Ma, Yunfei Qiu, Li Fang, Huifang Shen, Wei Yao
TL;DR
This work tackles cross-domain hyperspectral image classification under limited labeling by proposing a self-supervised framework that learns transferable spectral-spatial representations. It introduces S²Former, a dual-branch Transformer with bidirectional cross-attention, together with a Frequency-Domain Constraint to preserve high-frequency spectral details, and DAFT to align semantic trajectories during few-shot fine-tuning. The approach achieves strong cross-domain generalization across four public HSIs, outperforming several baselines and demonstrating notable improvements in boundary accuracy and class separation under data-scarce conditions. The results suggest that integrating spectral-spatial modeling with diffusion-based alignment and frequency-domain supervision can substantially improve robustness to domain shifts in hyperspectral analysis.
Abstract
Self-supervised learning has demonstrated considerable potential in hyperspectral representation, yet its application in cross-domain transfer scenarios remains under-explored. Existing methods, however, still rely on source domain annotations and are susceptible to distribution shifts, leading to degraded generalization performance in the target domain. To address this, this paper proposes a self-supervised cross-domain transfer framework that learns transferable spectral-spatial joint representations without source labels and achieves efficient adaptation under few samples in the target domain. During the self-supervised pre-training phase, a Spatial-Spectral Transformer (S2Former) module is designed. It adopts a dual-branch spatial-spectral transformer and introduces a bidirectional cross-attention mechanism to achieve spectral-spatial collaborative modeling: the spatial branch enhances structural awareness through random masking, while the spectral branch captures fine-grained differences. Both branches mutually guide each other to improve semantic consistency. We further propose a Frequency Domain Constraint (FDC) to maintain frequency-domain consistency through real Fast Fourier Transform (rFFT) and high-frequency magnitude loss, thereby enhancing the model's capability to discern fine details and boundaries. During the fine-tuning phase, we introduce a Diffusion-Aligned Fine-tuning (DAFT) distillation mechanism. This aligns semantic evolution trajectories through a teacher-student structure, enabling robust transfer learning under low-label conditions. Experimental results demonstrate stable classification performance and strong cross-domain adaptability across four hyperspectral datasets, validating the method's effectiveness under resource-constrained conditions.