E-RGB-D: Real-Time Event-Based Perception with Structured Light

TL;DR

Monochrome event cameras struggle to capture color and rapid RGB-D information in dynamic scenes. The authors integrate a DLP-based Adaptive Structured Light system with an event camera to achieve per-event color and depth without frame aggregation, enabling frameless, high-speed RGB-D sensing. Key contributions include per-event disparity lookups, flexible pattern design with CP/FR trade-offs, and real-time color-depth reconstruction achieving up to 1.4 kHz color and 4 kHz depth, demonstrated on a 640×480 Prophesee sensor with a TI LightCrafter 4500 projector. The work advances rapid, low-latency 3D perception for robotics and real-time 3D reconstruction, with open-source ROS-based tooling available at the project repository.

Abstract

Event-based cameras (ECs) have emerged as bio-inspired sensors that report pixel brightness changes asynchronously, offering unmatched speed and efficiency in vision sensing. Despite their high dynamic range, temporal resolution, low power consumption, and computational simplicity, traditional monochrome ECs face limitations in detecting static or slowly moving objects and lack color information essential for certain applications. To address these challenges, we present a novel approach that integrates a Digital Light Processing (DLP) projector, forming Active Structured Light (ASL) for RGB-D sensing. By combining the benefits of ECs and projection-based techniques, our method enables the detection of color and the depth of each pixel separately. Dynamic projection adjustments optimize bandwidth, ensuring selective color data acquisition and yielding colorful point clouds without sacrificing spatial resolution. This integration, facilitated by a commercial TI LightCrafter 4500 projector and a monocular monochrome EC, not only enables frameless RGB-D sensing applications but also achieves remarkable performance milestones. With our approach, we achieved a color detection speed equivalent to 1400 fps and 4 kHz of pixel depth detection, significantly advancing the realm of computer vision across diverse fields from robotics to 3D reconstruction methods. Our code is publicly available: https://github.com/MISTLab/event_based_rgbd_ros

Figures (16)

• Figure 1: Color (left) and depth (right) detection of a volleyball thrown in front of a VGA monocular event-based camera, reconstructed at 120 fps from $\sim 1.5$ m using the proposed method.
• Figure 2: Color detection of a printed color wheel, as introduced by Bajestani_2023_WACV. Top: The proposed procedure involves projecting various light patterns in different wavelengths/colors. Bottom, from left to right: high-resolution Ground Truth (GT), reconstructed images captured by a VGA monochrome EC in the channels of Red, Green, Blue, and the fully reconstructed image.
• Figure 3: Proposed ASL pattern types for balancing speed and detail in E-RGB-D scanning. M is greater than N, meaning more points are reconstructed and have higher Coverage Percentage (CP). We did not use a pseudo-random dot pattern to detect the depth, but it could be used to add color to methods that detect only depth, such as those proposed by huang2021high.
• Figure 4: Pattern sequence and their exposure times in microseconds. In our experiments, we used mode 3 with two different values for $n$ (23 and 45), and mode 4 with $n$=23. One ID for each pattern type would be enough, but we could have different IDs for each color or depth pattern mode. While this increases the total scanning time, it makes the system more robust and trackable.
• Figure 5: Top Left: Output of the D455 RGB camera. Top Right: The RGB image reconstructed by the proposed method with a Monochrome EC. Middle Row: Depth detection comparison between D455 (Left), muglikar2021esl's work named ESL (Middle), and our method, E-RGB-D (Right). Bottom Row: Zoomed-in view of the middle row. Note that the color differences are due to defining different minimum and maximum values for the jet-coded colorization; the actual difference in mm is detailed in Section \ref{['results']}.