Robust Hyperspectral Image Panshapring via Sparse Spatial-Spectral Representation

TL;DR

This work tackles robust hyperspectral image pansharpening by fusing LRHSI with HRMSI through a sparse spatial-spectral representation. The proposed S$^{3}$RNet leverages a Multi-Branch Fusion Network to learn multi-scale spatial-spectral features, a Spatial-Spectral Attention Weight Block to enforce sparsity and adaptively weight informative features, and a Dense Feature Aggregation Block to promote efficient feature propagation. Empirical results on AVIRIS data show state-of-the-art reconstruction performance and strong robustness to noise, outperforming multiple CNN, transformer, and Mamba-based competitors. The framework offers practical benefits for remote sensing tasks where high spectral fidelity must be preserved under challenging imaging conditions, and the authors plan to release the code publicly.

Abstract

High-resolution hyperspectral imaging plays a crucial role in various remote sensing applications, yet its acquisition often faces fundamental limitations due to hardware constraints. This paper introduces S$^{3}$RNet, a novel framework for hyperspectral image pansharpening that effectively combines low-resolution hyperspectral images (LRHSI) with high-resolution multispectral images (HRMSI) through sparse spatial-spectral representation. The core of S$^{3}$RNet is the Multi-Branch Fusion Network (MBFN), which employs parallel branches to capture complementary features at different spatial and spectral scales. Unlike traditional approaches that treat all features equally, our Spatial-Spectral Attention Weight Block (SSAWB) dynamically adjusts feature weights to maintain sparse representation while suppressing noise and redundancy. To enhance feature propagation, we incorporate the Dense Feature Aggregation Block (DFAB), which efficiently aggregates inputted features through dense connectivity patterns. This integrated design enables S$^{3}$RNet to selectively emphasize the most informative features from differnt scale while maintaining computational efficiency. Comprehensive experiments demonstrate that S$^{3}$RNet achieves state-of-the-art performance across multiple evaluation metrics, showing particular strength in maintaining high reconstruction quality even under challenging noise conditions. The code will be made publicly available.

Figures (3)

• Figure 1: Normalized energy distribution of fused layer across 172 channels between the proposed S$^{3}$RNet, CNN-based methodMin2021MSSJFL, Transformer-based methodHyperTransformer, and Mamba-based methodFusionMamba. The majority of energy concentrate in head side, indicating our method's sparsity. By integrating spatial-spectral representation, our method have archived state-of-the-art performance with robustness against heavy noise compared with other peer-methods.
• Figure 2: Layer-wise CKA similarity analysis CKA. It indicates that different branches ($\mathbf{Q}$-$\mathbf{K}$-$\mathbf{V}$-$\mathbf{Z}$ branch) learn specialized feature representations. The low cross-branch similarity (dark regions) suggests each branch captures unique aspects of the spatial-spectral information, rather than redundancy.
• Figure 3: Architecture of the proposed S$^3$RNet for hyperspectral image pansharping. Multi-Branch Fusion Network (MBFN) for extracting multi-scale spatial-spectral features in $\mathbf{Q}$-$\mathbf{K}$-$\mathbf{V}$-$\mathbf{Z}$ branch, Spatial-Spectral Attention Weight Block (SSAWB) for adaptive feature refinement and improving robustness against noise interference with sparse feature representation, and Dense Feature Aggregation Block (DFAB) for efficient feature integration.