Event Spectroscopy: Event-based Multispectral and Depth Sensing using Structured Light

TL;DR

This work introduces an all-in-one event-based sensing system that fuses depth reconstruction and multispectral imaging using a single sensor and structured light. By leveraging an event camera with active, wavelength-tuned illumination, the approach achieves low-latency depth with high fidelity and competitive spectral accuracy compared to commercial multispectral systems. The authors demonstrate significant depth accuracy gains over conventional sensors, robust performance under varying illumination, and improved material differentiation in both indoor and real forest environments. The results suggest a promising path toward lightweight, integrated perception for UAVs operating in complex natural environments, with noted limitations and clear avenues for extending spectral bandwidth and UAV deployment.

Abstract

Uncrewed aerial vehicles (UAVs) are increasingly deployed in forest environments for tasks such as environmental monitoring and search and rescue, which require safe navigation through dense foliage and precise data collection. Traditional sensing approaches, including passive multispectral and RGB imaging, suffer from latency, poor depth resolution, and strong dependence on ambient light - especially under forest canopies. In this work, we present a novel event spectroscopy system that simultaneously enables high-resolution, low-latency depth reconstruction with integrated multispectral imaging using a single sensor. Depth is reconstructed using structured light, and by modulating the wavelength of the projected structured light, our system captures spectral information in controlled bands between 650 nm and 850 nm. We demonstrate up to $60\%$ improvement in RMSE over commercial depth sensors and validate the spectral accuracy against a reference spectrometer and commercial multispectral cameras, demonstrating comparable performance. A portable version limited to RGB (3 wavelengths) is used to collect real-world depth and spectral data from a Masoala Rainforest. We demonstrate the use of this prototype for color image reconstruction and material differentiation between leaves and branches using spectral and depth data. Our results show that adding depth (available at no extra effort with our setup) to material differentiation improves the accuracy by over $30\%$ compared to color-only method. Our system, tested in both lab and real-world rainforest environments, shows strong performance in depth estimation, RGB reconstruction, and material differentiation - paving the way for lightweight, integrated, and robust UAV perception and data collection in complex natural environments.

Figures (7)

• Figure 1: Event Spectroscopy: We propose an all-in-one solution for depth sensing, color image reconstruction and multispectral sensing. Usable for instance, for an uncrewed aerial vehicle (UAV) navigating in forest environments (C). A) Our portable setup consisting of an event camera and a projector as an illumination source B) Generated depth of scene acquired using structured light, D) spectral image reconstructed using events, E) material segmentation of leaves using spectral and depth data.
• Figure 2: The simulation demonstrates how a change in the source follower bandwidth can regulate the observed signal amplitude downstream. Responses for different follower gains are shown as dotted lines.
• Figure 3: The full spectrum lab illumination setup with our event-camera setup.
• Figure 4: Comparing depth accuracy of different sensors when imaging challenging scenes such as thin and hollow structures or strong reflections.
• Figure 5: Samples of reconstructed color-corrected images with ground-truth (left), ours (center), and using Event Counting (right)Ehsan2022,