Unleashing Correlation and Continuity for Hyperspectral Reconstruction from RGB Images

TL;DR

This work tackles the ill-posed task of reconstructing hyperspectral images from RGB inputs by explicitly modeling two core inter-spectral properties: local correlation among nearby spectral bands and global continuity across the spectral range. The Correlation and Continuity Network (CCNet) introduces three key components—Group-wise Spectral Correlation Modeling (GrSCM) for local band similarity, Neighborhood-wise Spectral Continuity Modeling (NeSCM) for global progressive variation, and Patch-wise Adaptive Fusion (PAF) to fuse these features in a region-aware manner—along with an effective loss combining $L_{MRAE}$ and a spectral difference term. The approach yields state-of-the-art performance on NTIRE2022 and NTIRE2020 datasets, with ablations confirming the importance of both GrSCM and NeSCM and the benefits of patch-wise fusion. By enabling accurate, high-fidelity HSI reconstruction from low-cost RGB imagery, CCNet holds significant potential for scalable hyperspectral imaging in remote sensing, environmental monitoring, and mineral identification.

Abstract

Reconstructing Hyperspectral Images (HSI) from RGB images can yield high spatial resolution HSI at a lower cost, demonstrating significant application potential. This paper reveals that local correlation and global continuity of the spectral characteristics are crucial for HSI reconstruction tasks. Therefore, we fully explore these inter-spectral relationships and propose a Correlation and Continuity Network (CCNet) for HSI reconstruction from RGB images. For the correlation of local spectrum, we introduce the Group-wise Spectral Correlation Modeling (GrSCM) module, which efficiently establishes spectral band similarity within a localized range. For the continuity of global spectrum, we design the Neighborhood-wise Spectral Continuity Modeling (NeSCM) module, which employs memory units to recursively model the progressive variation characteristics at the global level. In order to explore the inherent complementarity of these two modules, we design the Patch-wise Adaptive Fusion (PAF) module to efficiently integrate global continuity features into the spectral features in a patch-wise adaptive manner. These innovations enhance the quality of reconstructed HSI. We perform comprehensive comparison and ablation experiments on the mainstream datasets NTIRE2022 and NTIRE2020 for the spectral reconstruction task. Compared to the current advanced spectral reconstruction algorithms, our designed algorithm achieves State-Of-The-Art (SOTA) performance.

Figures (6)

• Figure 1: The process of spectral reconstruction from RGB image. The RGB image has three channels: R (red), G (green), and B (blue), which are perceivable by the human eye. The hyperspectral image composed of multiple bands, and the images from left to right represent the visualization of response values from low to high across different wavelength.
• Figure 2: The overall structure of the CCNet, which primarily consists of $m$ Spectrum Reconstruction Units based on the U-Net model ronneberger2015u. The Spectrum Reconstruction Unit consists mainly of multiple CSRM Blocks, with the core components of each CSRM block being Inter-Spectral Relationship Modeling module and Intra-Spectral Spatiality Modeling module.
• Figure 3: The overall structure of NeSCM module, where the blue dashed lines and red dashed lines represent the forward branch and the backward branch, respectively. Each branch is composed of multiple CMUs.
• Figure 4: The overall structure of PAF module. The similarity calculation in the orange rectangle box means that a $F_{att}^n(i)$ is calculated once with all $F_{cp}^n$, where $F_{cp}^n\in\{F_{cp}^n(1),F_{cp}^n(2),\dots, F_{cp}^n(k)\}$
• Figure 5: The visualized $L_1$ reconstruction errors of different spectral reconstruction algorithms on different samples. The red area in the figure indicates a large error, and the blue area indicates a small error. For these samples, we randomly selected one point in each image, specifically marked by red circles in the RGB images, and visualized the spectral response curves, as shown in Fig. \ref{['compare_line']}.