Grayscale to Hyperspectral at Any Resolution Using a Phase-Only Lens

TL;DR

The paper demonstrates that hyperspectral images with 31 spectral channels can be reconstructed from a single grayscale snapshot captured through a phase-only metalens, by training a patch-based conditional diffusion model and enforcing global optical consistency via PSF guidance. The method processes patches in parallel, learns strong local priors, and uses guided sampling to stitch patches into a measurement-consistent full HSI while providing per-pixel uncertainty through multiple samples. It shows state-of-the-art performance against baselines in grayscale-to-HSI tasks, robust generalization across datasets and resolutions, and reveals interpretable connections between PSF structure and spectral reconstruction. This approach opens a path toward compact, light-efficient snapshot hyperspectral cameras that rely on a single, phase-only lens and minimal sensing hardware.

Abstract

We consider the problem of reconstructing a HxWx31 hyperspectral image from a HxW grayscale snapshot measurement that is captured using only a single diffractive optic and a filterless panchromatic photosensor. This problem is severely ill-posed, but we present the first model that produces high-quality results. We make efficient use of limited data by training a conditional denoising diffusion model that operates on small patches in a shift-invariant manner. During inference, we synchronize per-patch hyperspectral predictions using guidance derived from the optical point spread function. Surprisingly, our experiments reveal that patch sizes as small as the PSFs support achieve excellent results, and they show that local optical cues are sufficient to capture full spectral information. Moreover, by drawing multiple samples, our model provides per-pixel uncertainty estimates that strongly correlate with reconstruction error. Our work lays the foundation for a new class of high-resolution snapshot hyperspectral imagers that are compact and light-efficient.

Figures (14)

• Figure 1: The RGB-projection and two representative spectra of a hyperspectral image (HSI) reconstructed from the chromatic aberration encoded in a grayscale measurement. Patches of the $1280\times 1280$ measurement are processed in parallel using guided diffusion, and the reconstructed HSI is sampled several times to compute uncertainty. Graphs show model outputs (green), ablated outputs without guidance (magenta), and ground truth (black).
• Figure 2: (a) A hyperspectral scene is imaged through a diffractive lens, producing an optically-coded measurement on a filterless photosensor. (b) For a single hyperspectral patch ($64\times64\times31$, red square), the point-spread function (PSF, $32\times32\times31$) induces a distinct blur and shift at each wavelength. The measurement patch is a sum and crop over wavelengths, while some signal is scattered outside the patch onto neighboring patches.
• Figure 3: PSFs (middle row, projected to RGB) used in our experiments. For context, we show the ideal achromatic PSF (left) and an example measurement for each PSF (bottom row). From left to right, measurements become blurrier. Each PSF is induced by a flat optic with a particular nanocylinder radii pattern (top row).
• Figure 4: Reconstructing a full-field HSI from an input measurement is achieved by splitting the measurement into patches. Each patch is concatenated with a noise sample $\mathbf{x}_{T}^{(i)}$ and then denoised to obtain an intermediate prediction $\hat{\mathbf{x}}_0^{(i)}$. Guidance is provided by stitching these predictions into a full-field HSI, applying the spectral PSF via convolution and summation, and comparing the result with the input measurement. The guidance gradient is used to update all patches with a reverse diffusion step, and this process is iteratively repeated. Pixel-wise uncertainty is estimated by performing multiple samplings with different random seeds.
• Figure 5: Grayscale-to-HSI reconstructions. Our estimate and the True HSIs are projected to RGB. Graphs display two spectral profiles at pixel marked in red. Bold green is our model's mean spectral estimate and fill displays uncertainty. Predictions from the three next-best comparison models are shown in red.