Spectral-wise Implicit Neural Representation for Hyperspectral Image Reconstruction

TL;DR

This work tackles the rigidity of spectral reconstruction in CASSI systems by introducing Spectral-wise Implicit Neural Representation (SINR), a continuous-spectrum INR framework. SINR leverages spectral-wise attention, Fourier coordinate encoding, and a spectral scale factor to model HSI as a continuous function of spectral coordinates, enabling arbitrary spectral magnification with a single trained model. It integrates an encoder to produce latent spectral codes and an implicit decoder to generate high-resolution spectra, showing superior spectral fidelity and edge detail across multiple datasets compared to fixed-band and interpolation baselines. The approach significantly enhances the practicality of CASSI by eliminating the need to retrain for different spectral band counts and opens avenues for spatial-spectral continuous reconstruction in hyperspectral imaging.

Abstract

Coded Aperture Snapshot Spectral Imaging (CASSI) reconstruction aims to recover the 3D spatial-spectral signal from 2D measurement. Existing methods for reconstructing Hyperspectral Image (HSI) typically involve learning mappings from a 2D compressed image to a predetermined set of discrete spectral bands. However, this approach overlooks the inherent continuity of the spectral information. In this study, we propose an innovative method called Spectral-wise Implicit Neural Representation (SINR) as a pioneering step toward addressing this limitation. SINR introduces a continuous spectral amplification process for HSI reconstruction, enabling spectral super-resolution with customizable magnification factors. To achieve this, we leverage the concept of implicit neural representation. Specifically, our approach introduces a spectral-wise attention mechanism that treats individual channels as distinct tokens, thereby capturing global spectral dependencies. Additionally, our approach incorporates two components, namely a Fourier coordinate encoder and a spectral scale factor module. The Fourier coordinate encoder enhances the SINR's ability to emphasize high-frequency components, while the spectral scale factor module guides the SINR to adapt to the variable number of spectral channels. Notably, the SINR framework enhances the flexibility of CASSI reconstruction by accommodating an unlimited number of spectral bands in the desired output. Extensive experiments demonstrate that our SINR outperforms baseline methods. By enabling continuous reconstruction within the CASSI framework, we take the initial stride toward integrating implicit neural representation into the field.

Figures (9)

• Figure 1: Correlation and Complementarity among HSI Spectral Channels Spatial information in adjacent spectral channels displays significant correlation, as evidenced by the similarity between 2D images rendered at 450.0$\rm {nm}$ and 472.2$\rm {nm}$. Conversely, spatial information with substantial wavelength differences complements each other. This is evident in the corresponding 2D images at 450.0$\rm {nm}$ and 650.0$\rm {nm}$, which capture distinct portions of the RGB image's spatial structure.
• Figure 2: The flowchart of CASSI system. Other optical parts are eliminated in favor of a 2D-coded aperture and a dispersion of the CASSI system. For synthetic HSI cubes, measurements can be mathematically computed by modeling the CASSI system.
• Figure 3: Overall network architecture of continuous spectral reconstruction, which comprises an encoder and our Spectral-wise Implicit Neural Representation (SINR). Inputs of SINR are as follows: scale factor $\boldsymbol S$, feature map extracted from an encoder $\boldsymbol Z$, and spectral coordinates $\boldsymbol X$.
• Figure 4: The flowchart of the proposed method SINR.
• Figure 5: Comparisons of reconstruction results on ICVL. SINR enables the reconstructed image to have sharper edges. The right part is the relative spectrum of points a, b, c, and d. The reconstruction spectrum of our method (the purple line) is closer to the ground truth (the red line). Corr refers to the correlation coefficient between the spectral curve of the reconstructed image and the ground truth.