Dense Dispersed Structured Light for Hyperspectral 3D Imaging of Dynamic Scenes

TL;DR

Hyperspectral 3D imaging of dynamic scenes has been hampered by slow acquisition and bulky, expensive setups. This work presents Dense Dispersed Structured Light (DDSL), a compact active-stereo approach using an RGB projector with a diffraction grating to project spectrally dispersed patterns, coupled with stereo RGB cameras to recover depth and hyperspectral data from a small number of projections. The authors design spectrally multiplexed DDSL patterns, formulate a dispersion-aware image formation model, and develop a reconstruction pipeline that yields depth accuracy around 4 mm and spectral FWHM of 15.5 nm at 6.6 fps. The method demonstrates superior speed and accuracy over prior affordable hyperspectral 3D imaging methods, enabling practical analysis of dynamic materials and scenes in real-world settings.

Abstract

Hyperspectral 3D imaging captures both depth maps and hyperspectral images, enabling comprehensive geometric and material analysis. Recent methods achieve high spectral and depth accuracy; however, they require long acquisition times often over several minutes or rely on large, expensive systems, restricting their use to static scenes. We present Dense Dispersed Structured Light (DDSL), an accurate hyperspectral 3D imaging method for dynamic scenes that utilizes stereo RGB cameras and an RGB projector equipped with an affordable diffraction grating film. We design spectrally multiplexed DDSL patterns that significantly reduce the number of required projector patterns, thereby accelerating acquisition speed. Additionally, we formulate an image formation model and a reconstruction method to estimate a hyperspectral image and depth map from captured stereo images. As the first practical and accurate hyperspectral 3D imaging method for dynamic scenes, we experimentally demonstrate that DDSL achieves a spectral resolution of 15.5 nm full width at half maximum (FWHM), a depth error of 4 mm, and a frame rate of 6.6 fps.

Figures (9)

• Figure 1: We introduce a spectrally multiplexed Dense Dispersed Structured Light (DDSL), accurate hyperspectral 3D imaging method for dynamic scenes. (a) Capture configuration, (b) estimated hyperspectral image in sRGB and depth image for dynamic scenes, (c) estimated hyperspectral image, (d) comparison with spectroradiometer measurements.
• Figure 2: Imaging System. (a) Our active stereo system comprises RGB stereo cameras and a RGB projector equipped with a diffraction grating. (b) The diffraction grating introduces dispersion to the projector light. (c) Spectral sensitivity and emission functions of the camera and the projector. (d) Diffraction grating efficiency.
• Figure 3: Image Formation. (a) Light transport of the dispersed light projection of the mapping function $\psi$. (b) Sub-pixel accurate sample collection for data-driven backward modeling. Calibrated backward mapping model that relates pixel point to projector horizontal position (c) for depth given a specific wavelength and (d) for wavelength given a fixed depth value.
• Figure 4: DDSL Pattern Designs. (a) Three parameters of DDSL patterns. We visualize the issues raised from large or small values for each parameter (b) line offset, (c) line shift, (d) line width. Refer to the text for details. (e) We project DDSL patterns and a single black pattern, and the captured images under such patterns are spectrally multiplexed. (f) Illuminated wavelengths for DDSL and black patterns onto the scene point.
• Figure 5: Black Optical Flow. We warp adjacent frame image to a target frame using naive optical flow under DDSL patterns and our black optical flow method to show the effectiveness of our method. We estimate optical flows using pretrained RAFT lipson2021raft network for both methods. Note that the target frame and adjacent frames are captured under different DDSL patterns. Therefore, the evaluation should primarily focus on geometric alignment rather than color consistency. (a) Warped images based on naive optical flow and black optical flow. (b) Visualization of the interpolated optical flows using the black optical flow.