M3SR: Multi-Scale Multi-Perceptual Mamba for Efficient Spectral Reconstruction

TL;DR

Problem: Efficiently reconstruct HSIs from RGB images, addressing limited spatial perception and single-scale features. Approach: A multi-scale, multi-perceptual Mamba architecture (M3SR) that embeds a Multi-Perceptual Fusion (MPF) block into a U-Net to fuse global, intermediate, and local features across spatial, frequency, and spectral domains. Contributions: design of the MPF block, integration with a U-Net for multi-scale SR, and comprehensive experiments showing superior accuracy with reduced parameters and FLOPs on NTIRE2022, NTIRE2020-Clean/Realworld, and CAVE. Significance: provides a practical, efficient solution for hyperspectral reconstruction with potential for real-world deployment in environmental, medical, and agricultural imaging.

Abstract

The Mamba architecture has been widely applied to various low-level vision tasks due to its exceptional adaptability and strong performance. Although the Mamba architecture has been adopted for spectral reconstruction, it still faces the following two challenges: (1) Single spatial perception limits the ability to fully understand and analyze hyperspectral images; (2) Single-scale feature extraction struggles to capture the complex structures and fine details present in hyperspectral images. To address these issues, we propose a multi-scale, multi-perceptual Mamba architecture for the spectral reconstruction task, called M3SR. Specifically, we design a multi-perceptual fusion block to enhance the ability of the model to comprehensively understand and analyze the input features. By integrating the multi-perceptual fusion block into a U-Net structure, M3SR can effectively extract and fuse global, intermediate, and local features, thereby enabling accurate reconstruction of hyperspectral images at multiple scales. Extensive quantitative and qualitative experiments demonstrate that the proposed M3SR outperforms existing state-of-the-art methods while incurring a lower computational cost.

Figures (5)

• Figure 1: PSNR v.s. Parameters v.s. FLOPs comparison with existing spectral reconstruction methods. For an intuitive analysis, FLOPs and PSNR are represented by the horizontal and vertical axis, and the circle radius indicates parameters. The proposed M3SR achieves a better balance between PSNR and parameters and FLOPs.
• Figure 2: Input image and sub-bands after discrete wavelet transform decomposition.
• Figure 3: The architecture of our M3SR and the structure of the multi-perceptual fusion (MPF) block.
• Figure 4: The structures of VSS block and Mamba block.
• Figure 5: Comparison of reconstruction results on the NTIRE2022 dataset using input image ARAD_1K_0907 (630 nm).