Lumosaic: Hyperspectral Video via Active Illumination and Coded-Exposure Pixels

Abstract

We present Lumosaic, a compact active hyperspectral video system designed for real-time capture of dynamic scenes. Our approach combines a narrowband LED array with a coded-exposure-pixel (CEP) camera capable of high-speed, per-pixel exposure control, enabling joint encoding of scene information across space, time, and wavelength within each video frame. Unlike passive snapshot systems that divide light across multiple spectral channels simultaneously and assume no motion during a frame's exposure, Lumosaic actively synchronizes illumination and pixel-wise exposure, improving photon utilization and preserving spectral fidelity under motion. A learning-based reconstruction pipeline then recovers 31-channel hyperspectral (400-700 nm) video at 30 fps and VGA resolution, producing temporally coherent and spectrally accurate reconstructions. Experiments on synthetic and real data demonstrate that Lumosaic significantly improves reconstruction fidelity and temporal stability over existing snapshot hyperspectral imaging systems, enabling robust hyperspectral video across diverse materials and motion conditions.

Figures (19)

• Figure 1: (A) Lumosaic is a compact active hyperspectral video system that combines programmable narrowband illumination with pixel-wise coded exposure to jointly encode spatial, spectral, and temporal information within each video frame, enabling real-time capture at 30 fps. (B) Reconstructed frames of a rotating globe, rendered in sRGB, showing smooth temporal progression despite rapid motion. (C) Reconstructed spectral channels (400-700 nm, 10 nm intervals) from the highlighted frame.
• Figure 2: The operation of a single pixel in a coded-exposure-pixel (CEP) camera. Each pixel alternates between two charge storage sites (Bucket 0 and Bucket 1) according to a binary exposure code. During each sub-frame, only one bucket integrates incident light, enabling temporal multiplexing across the exposure period.
• Figure 2: The hardware schematic of Lumosaic's active illumination module. The ESP32 microcontroller coordinates the LED driver array and issues synchronization pulses to the CEP camera.
• Figure 3: (A) Lumosaic features a CEP camera synchronized with a programmable array of narrowband LEDs. (B) Each sub-frame is assigned a unique LED and spatial exposure mask, producing a dense spatio-spectro-temporal scene encoding within a single video frame.
• Figure 3: The $3\times4$ mosaic tile that forms the basis of Lumosaic's spatial-spectral coding.