Spectrum from Defocus: Fast Spectral Imaging with Chromatic Focal Stack

TL;DR

SfD introduces a chromatic focal sweep hyperspectral camera built from two off-the-shelf lenses and a grayscale sensor, enabling rapid reconstruction of high-quality spectral data from a compact, light-efficient system. The core is a physics-based inverse algorithm (plug-and-play ADMM) that demixes, deconvolves, and denoises a grayscale focal stack, leveraging a low-dimensional spectral prior and a fast inversion exploiting structure in the forward model. Placing emphasis on optical simplicity and interpretability, SfD achieves state-of-the-art reconstruction quality with substantially faster compute than many prior methods, while mitigating hallucination common in data-driven approaches. The approach demonstrates strong performance in synthetic and real scenes, with robust spectral fidelity under photon-limited conditions and promising potential for fast, compact hyperspectral imaging in practical applications.

Abstract

Hyperspectral cameras face harsh trade-offs between spatial, spectral, and temporal resolution in an inherently low-photon regime. Computational imaging systems break through these trade-offs with compressive sensing, but require complex optics and/or extensive compute. We present Spectrum from Defocus (SfD), a chromatic focal sweep method that recovers state-of-the-art hyperspectral images with a small system of off-the-shelf optics and < 1 second of compute. Our camera uses two lenses and a grayscale sensor to preserve nearly all incident light in a chromatically-aberrated focal stack. Our physics-based iterative algorithm efficiently demixes, deconvolves, and denoises the blurry grayscale focal stack into a sharp spectral image. The combination of photon efficiency, optical simplicity, and physical modeling makes SfD a promising solution for fast, compact, interpretable hyperspectral imaging.

Figures (6)

• Figure 1: Comparison and overview of our Spectrum from Defocus (SfD) camera. (a) Our camera provides state-of-the-art PSNR with simple optics and a high frame rate, which we compute as $1/(t_{\text{exposure}} + t_{\text{compute}})$ with a simulated 5 sec total exposure for each method. Bubble diameters correspond to the number of optical components in each method. See Table \ref{['tab:comparison']} for details. (b) Our hardware prototype uses a moving lens to change the focus of chromatically aberrated light on its way to a grayscale sensor. (c) Our imaging system captures a grayscale focal stack and reconstructs images in under a second. We show an RGB projection of a 12-channel recovered image, with a high-resolution inset (blue square) and reconstructed spectrum (red star).
• Figure 2: Optical design. (a) The system consists of a lens pair in which the second lens is moved to $N$ positions, $z_1, z_2, \cdots, z_N$, respectively when capturing $N$ images of a scene. We assume the imaging target is within the depth of field throughout this paper. The $N$ captured images, $I_1,\cdots, I_N$, individually focus $N$ discrete, predetermined wavelengths, $\lambda_1,\cdots, \lambda_N$, while blurring others, forming a chromatic focal stack. (b) Measured point spread functions (PSFs) of the real-world prototype at different wavelengths $\lambda_i$ and lens position $z_i$. The PSFs clearly show the chromatic focal shift , as the focus shifts to different wavelengths at different lens positions $z_i$. Although six different lens positions are shown, only a subset of lens positions is used during reconstruction, corresponding to the number and location of captured measurements.
• Figure 3: Comparison of simulated SOTA methods. We illustrate the performance of several SOTA methods simulated on five hyperspectral images from the KAIST choi2017high and CAVE yasuma2010generalized dataset. The PSNR of the reconstructed hyperspectral image is listed for each, below an RGB projection of the data. Our method, using a simulated focal stack of 5 measurements, performs well, with high spatial resolution and no color tinting (compare to MST). Spectra are shown for the numbered points, indicated with white arrows on each image. We reconstruct spectra smoothly and accurately. Note that MST can only reconstruct a limited wavelength range, as its training data spans only $453-648$ nm.
• Figure 4: Quantitative evaluation on real data. We reconstruct the Macbeth color chart from a 5-measurement focal stack to evaluate hyperspectral and trichromatic reconstruction quality. The spectral reconstruction of the full image with three spectra from RGB patches and the $\Delta E_{00}$ perceptual error of each square are presented. The SAM in degree is also reported for the RGB color patches.
• Figure 5: Robustness of Color Checker Recovery. (a) We assess the reconstructed spectral accuracy at varying distances from the calibration plane. Compared to the theoretical working distance of $16$ cm from the calibration plane, mild deviations from this range do not significantly impact the results. (b) We evaluate spectral accuracy as a function of the number of measurements. Adding measurements (which in the real data setting corresponds to a longer total exposure time), improves reconstruction, with performance saturating more quickly in RGB than in spectrum. See supplement for corresponding images.