Limitations of Data-Driven Spectral Reconstruction -- An Optics-Aware Analysis

TL;DR

This work scrutinizes data-driven RGB-to-spectral reconstruction through an optics-aware lens, revealing that conventional datasets and evaluation metrics obscure fundamental limitations. It demonstrates atypical overfitting, metameric failures, and the potential of optical aberrations and engineered spectral encodings to partially mitigate these issues. The study introduces metameric augmentation and aberration-aware training as practical steps, but emphasizes that robust snapshot spectral imaging ultimately requires larger, more diverse datasets and deliberate optical encoding strategies. The findings highlight the need to rethink problem formulations and integrate physical imaging constraints to achieve reliable spectral recovery from RGB inputs.

Abstract

Hyperspectral imaging empowers machine vision systems with the distinct capability of identifying materials through recording their spectral signatures. Recent efforts in data-driven spectral reconstruction aim at extracting spectral information from RGB images captured by cost-effective RGB cameras, instead of dedicated hardware. Published work reports exceedingly high numerical scores for this reconstruction task, yet real-world performance lags substantially behind. We systematically analyze the performance of such methods. First, we evaluate the overfitting limitations with respect to current datasets by training the networks with less data, validating the trained models with unseen yet slightly modified data and cross-dataset validation. Second, we reveal fundamental limitations in the ability of RGB to spectral methods to deal with metameric or near-metameric conditions, which have so far gone largely unnoticed due to the insufficiencies of existing datasets. We validate the trained models with metamer data generated by metameric black theory and re-training the networks with various forms of metamers. This methodology can also be used for data augmentation as a partial mitigation of the dataset issues, although the RGB to spectral inverse problem remains fundamentally ill-posed. Finally, we analyze the potential for modifying the problem setting to achieve better performance by exploiting optical encoding provided by either optical aberrations or deliberate optical design. Our experiments show such approaches provide improved results under certain circumstances, but their overall performance is limited by the same dataset issues. We conclude that future progress on snapshot spectral imaging will heavily depend on the generation of improved datasets which can then be used to design effective optical encoding strategies. Code: https://github.com/vccimaging/OpticsAwareHSI-Analysis.

Figures (9)

• Figure 1: An example scene fake_and_real_food_ms from the CAVE dataset yasuma2010generalized consists of objects with visually similar colors, but actually different spectra. Left: Color image with highlighted points on the red peppers. Their RGB values are nearly the same. Right: Ground-truth and reconstructed spectra at the corresponding points show their spectral differences. The reconstructed spectra are predicted by the pre-trained MST++ model cai2022mst++ on the ARAD1K dataset arad2022ntire. The neural network struggles to distinguish either the two spectra from each other, or from their true spectra.
• Figure 2: Spectral image formation models used in the analysis in this work. Top: In the NTIRE spectral recovery challenges, an RGB image is considered as a linear projection from a high-dimensional hyperspectral datacube to a 3D color image. The existence of metamerism results in identical RGB images for different spectra. The neural network trained in this way cannot distinguish their corresponding spectra. Bottom: A possible mitigation to this problem is to include the optical aberrations of the lens in the image formation model. Spectral information is encoded into the aberrated RGB images, enabling the neural network to tell the difference between metamers. In both cases, the RGB image differences are shown on the right (intensity enhanced for better visualization).
• Figure 3: Validation performance for MST++ cai2022mst++ with 100%, 50%, and 20% of the original training data on ARAD1K arad2022ntire.
• Figure 4: Validation with metamers for MST++ cai2022mst++. An example Scene ARAD_1K_0944 is shown to visualize the standard and metamer datacubes. Top left: the standard and metamer data result in similar color images. Bottom left: ground-truth and reconstructed spectra from two labeled points. Right: ground-truth and reconstructed spectral images in 420 nm, 500 nm, 550 nm, 580 nm, and 660 nm.
• Figure 5: Training MST++ with metamers. It fails to combat fixed metamers and on-the-fly metamers, in particular on the spectral accuracy SAM.