Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GMSR:Gradient-Guided Mamba for Spectral Reconstruction from RGB Images

Xinying Wang, Zhixiong Huang, Sifan Zhang, Jiawen Zhu, Paolo Gamba, Lin Feng

TL;DR

This work addresses spectral reconstruction from RGB images by balancing accuracy and efficiency with a lightweight Mamba-based backbone. GMSR-Net uses Gradient Mamba blocks in a tri-branch GM design supplemented by spatial and spectral gradient attention to maintain global context while preserving local structure. The approach achieves state-of-the-art results on NTIRE2020-Clean, CAVE, and Harvard with substantial reductions in parameters and FLOPs, demonstrating strong practical potential for in-camera or resource-constrained deployments. The combination of a global receptive field, linear complexity, and gradient-guided cues provides a robust framework for SR that advances both performance and efficiency in spectral restoration from RGB inputs.

Abstract

Mainstream approaches to spectral reconstruction (SR) primarily focus on designing Convolution- and Transformer-based architectures. However, CNN methods often face challenges in handling long-range dependencies, whereas Transformers are constrained by computational efficiency limitations. Recent breakthroughs in state-space model (e.g., Mamba) has attracted significant attention due to its near-linear computational efficiency and superior performance, prompting our investigation into its potential for SR problem. To this end, we propose the Gradient-guided Mamba for Spectral Reconstruction from RGB Images, dubbed GMSR-Net. GMSR-Net is a lightweight model characterized by a global receptive field and linear computational complexity. Its core comprises multiple stacked Gradient Mamba (GM) blocks, each featuring a tri-branch structure. In addition to benefiting from efficient global feature representation by Mamba block, we further innovatively introduce spatial gradient attention and spectral gradient attention to guide the reconstruction of spatial and spectral cues. GMSR-Net demonstrates a significant accuracy-efficiency trade-off, achieving state-of-the-art performance while markedly reducing the number of parameters and computational burdens. Compared to existing approaches, GMSR-Net slashes parameters and FLOPS by substantial margins of 10 times and 20 times, respectively. Code is available at https://github.com/wxy11-27/GMSR.

GMSR:Gradient-Guided Mamba for Spectral Reconstruction from RGB Images

TL;DR

This work addresses spectral reconstruction from RGB images by balancing accuracy and efficiency with a lightweight Mamba-based backbone. GMSR-Net uses Gradient Mamba blocks in a tri-branch GM design supplemented by spatial and spectral gradient attention to maintain global context while preserving local structure. The approach achieves state-of-the-art results on NTIRE2020-Clean, CAVE, and Harvard with substantial reductions in parameters and FLOPs, demonstrating strong practical potential for in-camera or resource-constrained deployments. The combination of a global receptive field, linear complexity, and gradient-guided cues provides a robust framework for SR that advances both performance and efficiency in spectral restoration from RGB inputs.

Abstract

Mainstream approaches to spectral reconstruction (SR) primarily focus on designing Convolution- and Transformer-based architectures. However, CNN methods often face challenges in handling long-range dependencies, whereas Transformers are constrained by computational efficiency limitations. Recent breakthroughs in state-space model (e.g., Mamba) has attracted significant attention due to its near-linear computational efficiency and superior performance, prompting our investigation into its potential for SR problem. To this end, we propose the Gradient-guided Mamba for Spectral Reconstruction from RGB Images, dubbed GMSR-Net. GMSR-Net is a lightweight model characterized by a global receptive field and linear computational complexity. Its core comprises multiple stacked Gradient Mamba (GM) blocks, each featuring a tri-branch structure. In addition to benefiting from efficient global feature representation by Mamba block, we further innovatively introduce spatial gradient attention and spectral gradient attention to guide the reconstruction of spatial and spectral cues. GMSR-Net demonstrates a significant accuracy-efficiency trade-off, achieving state-of-the-art performance while markedly reducing the number of parameters and computational burdens. Compared to existing approaches, GMSR-Net slashes parameters and FLOPS by substantial margins of 10 times and 20 times, respectively. Code is available at https://github.com/wxy11-27/GMSR.
Paper Structure (21 sections, 7 equations, 9 figures, 3 tables)

This paper contains 21 sections, 7 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Spectral Reconstruction from RGB Images.
  4. State Space Models
  5. Proposed Method
  6. Overall Architecture
  7. VSS block
  8. Spatial Gradient Attention
  9. Spectral Gradient Attention
  10. Loss Function
  11. Experiments
  12. Experimental Settiing
  13. Datasets
  14. Evaluation Metrics
  15. Implementation Details
  16. ...and 6 more sections

Figures (9)

  • Figure 1: PSNR $v.s.$ Params $v.s.$ FLOPs comparisons with existing spectral reconstruction approaches. For an intuitive analysis, FLOPs and PSNR are represented by the horizontal axis and vertical axis, and the circle radius indicates Params. The proposed Gradient-Guided Mamba (GMSR-Net) outperforms counterparts with dramatically lower FLOPs and Params requirements.
  • Figure 2: Overview of GMSR-Net. (a) GMSR-Net.(b) Gradient Mamba. (c) VSS Block. (d) Spatial Gradient Attention. (e) Spectral Gradient Attention.
  • Figure 3: Diagram for the 2D-Selective-Scan (SS2D) Operation.
  • Figure 4: Reconstruction results comparison on the NTIRE2020-Clean. The first row is ARAD_0463 (520 nm) and the second row is ARAD_0457 (490 nm).
  • Figure 5: Reconstruction results of different methods on the CAVE dataset. The first row is stuffe_toys (560 nm), the second row is superballs (680 nm).
  • ...and 4 more figures