Hyperspectral Reconstruction of Skin Through Fusion of Scattering Transform Features

TL;DR

The paper tackles reconstructing hyperspectral skin imagery from RGB plus a near-infrared band to overcome practical spectral-device limitations. It introduces a scattering-transform–based pipeline that operates in feature space, using two parallel matching networks for the even and odd HSI channels, followed by inverse networks and a MISR module to refine skin spectra. The approach achieves competitive spectral-angle mapper performance, with the best model (MISR 60 epochs) attaining an average SAM of 0.1179 ± 0.0129, close to a baseline MST++ method. By leveraging predefined wavelet filters and a two-layer scattering representation, the method reduces spectral-structure complexity and points to PCA-based simplification and deeper scattering as promising future directions, with significant potential for practical skin analysis applications.

Abstract

Hyperspectral imagery (HSI) is an established technique with an array of applications, but its use is limited due to both practical and technical issues associated with spectral devices. The goal of the ICASSP 2024 'Hyper-Skin' Challenge is to extract skin HSI from matching RGB images and an infrared band. To address this problem we propose a model using features of the scattering transform - a type of convolutional neural network with predefined filters. Our model matches and inverts those features, rather than the pixel values, reducing the complexity of matching while grouping similar features together, resulting in an improved learning process.