FRN: Fractal-Based Recursive Spectral Reconstruction Network

TL;DR

The paper tackles reconstructing hyperspectral images from RGB data, framing spectral reconstruction as a progressive, fractal-inspired process that predicts from broad to narrow bands. It introduces FRN, which recursively invokes atomic modules to exploit the intrinsic low-rank, self-similar structure of HSIs, and it pairs this with BAMamba, a band-aware state-space model that reduces computation via a band-wise mask. The approach delivers state-of-the-art reconstruction on the CAVE and Harvard datasets while using significantly fewer parameters, demonstrating both high accuracy and computational efficiency. This work has practical implications for affordable, rapid HSI acquisition from standard RGB cameras while maintaining strong spectral fidelity.

Abstract

Generating hyperspectral images (HSIs) from RGB images through spectral reconstruction can significantly reduce the cost of HSI acquisition. In this paper, we propose a Fractal-Based Recursive Spectral Reconstruction Network (FRN), which differs from existing paradigms that attempt to directly integrate the full-spectrum information from the R, G, and B channels in a one-shot manner. Instead, it treats spectral reconstruction as a progressive process, predicting from broad to narrow bands or employing a coarse-to-fine approach for predicting the next wavelength. Inspired by fractals in mathematics, FRN establishes a novel spectral reconstruction paradigm by recursively invoking an atomic reconstruction module. In each invocation, only the spectral information from neighboring bands is used to provide clues for the generation of the image at the next wavelength, which follows the low-rank property of spectral data. Moreover, we design a band-aware state space model that employs a pixel-differentiated scanning strategy at different stages of the generation process, further suppressing interference from low-correlation regions caused by reflectance differences. Through extensive experimentation across different datasets, FRN achieves superior reconstruction performance compared to state-of-the-art methods in both quantitative and qualitative evaluations.

Figures (9)

• Figure 1: PSNR-Parameters comparisons of FRN and SOTA methods.
• Figure 2: Overview of the Fractal-Based Progressive Spectral Reconstruction Paradigm: (a) illustrates how FRN reconstructs images at specific wavelengths in a coarse-to-fine manner, transitioning from wide spectral bands to narrow ones across multiple levels. (b) demonstrates the structural self-similarity of the modules within each level.
• Figure 3: The details of BAMamba. BAMamba is a U-Net style network built with state space models equipped with band-aware masks (BSSM). BSSM introduces a band-aware spatial mask that adaptively perceives the reflectance of objects at specific wavelengths, suppressing interference from pixels with lower correlation.
• Figure 4: Residual maps across the R, G, and B channels from a CAVE dataset sample.
• Figure 5: Comparison of the reconstruction results of different methods on one scene from the CAVE dataset, including seven SOTA methods and our FRN. We select three bands (20, 25, and 31) for visualization.