HyperColorization: Propagating spatially sparse noisy spectral clues for reconstructing hyperspectral images

TL;DR

A colorization algorithm to reconstruct hyperspectral images from a grayscale guide image and spatially sparse spectral clues is presented, demonstrating that the algorithm generalizes to varying spectral dimensions for hyperspectral images, and that colorizing in a low-rank space reduces compute time and the impact of shot noise.

Abstract

Hyperspectral cameras face challenging spatial-spectral resolution trade-offs and are more affected by shot noise than RGB photos taken over the same total exposure time. Here, we present a colorization algorithm to reconstruct hyperspectral images from a grayscale guide image and spatially sparse spectral clues. We demonstrate that our algorithm generalizes to varying spectral dimensions for hyperspectral images, and show that colorizing in a low-rank space reduces compute time and the impact of shot noise. To enhance robustness, we incorporate guided sampling, edge-aware filtering, and dimensionality estimation techniques. Our method surpasses previous algorithms in various performance metrics, including SSIM, PSNR, GFC, and EMD, which we analyze as metrics for characterizing hyperspectral image quality. Collectively, these findings provide a promising avenue for overcoming the time-space-wavelength resolution trade-off by reconstructing a dense hyperspectral image from samples obtained by whisk or push broom scanners, as well as hybrid spatial-spectral computational imaging systems.

Figures (13)

• Figure 1: HyperColorization simplified pipeline with an example image from brainard1998bear. We use a grayscale image to guide a whisk/push broom scanner or a computational imager to collect spatially sparse spectral samples. These noisy samples are used to estimate the best colorization dimension. Note that we show spatially uniform sampling at a rate of 3% but can see additional performance gains from image-guided sampling. Finally, the sparse samples are densified using an optimization-based spectral sample propagation algorithm. The code is available at https://github.com/NUBIVlab/HyperColorization and includes demos, documentation of each figure, and spectral results for every pixel in each image shown.
• Figure 2: Luminance differences between neighboring pixels carry information about spectral differences. The absolute difference between values on grayscale guide images (simulated by summing across visible spectral channels) is correlated with the L1 distance between their spectra. This correlation is strongest in (a), where the spectral differences are taken along the same wavelength range as the guide image. The relationship weakens in (b) where spectral bands not included in the guide image are considered, and is weaker still in (c) where there is no spectral overlap between the guide image's spectral bands and our target spectral bands. Data shown here is drawn from the dustbin and bulb images of the ICVL dataset arad2016sparse which includes 519 bands from 390 to 1043 nm.
• Figure 3: Luminance Adjustment. (a): Grayscale guide image, created by summing across wavelengths, with red dots indicating the uniformly-spaced locations of spectral clues (shown over 2$\times$2 pixels for visibility). (b): Color propagation does not preserve luminance information on hyperspectral images. (c): Renormalization of image (b) based on image (a) using Eq. \ref{['eq:lumbalance']} removes these artifacts. Image from chakrabarti2011statistics contains 31 channels from 400 to 700 nm and has been projected to RGB here for visualization.
• Figure 4: Singular vectors corresponding to the largest 6 singular values learned from the CAVE dataset yasuma2010generalized and their RGB projections for coefficients from -15 to +15. Note that RGB channels have been allowed to saturate.
• Figure 5: The relationship between noise level and the projection of hyperspectral clues onto our learned basis. The elbow location and vertical offset of these curves vary with noise and can be used to predict the highest-performing reconstruction dimensionality.