Event-Driven Dynamic Scene Depth Completion

TL;DR

Depth completion in dynamic scenes is hindered by rapid ego-motion and moving objects, causing RGB-D misalignment and local depth errors. EventDC introduces two motion-aware modules, EMA and LDF, that use high-temporal-resolution event streams to adapt sampling positions and refine depth around motion, respectively, and is guided by structure- and motion-aware losses. The authors also establish the first event-based depth completion benchmark with real and synthetic datasets, and demonstrate state-of-the-art performance across all benchmarks, including substantial improvements over existing RGB-D methods in RMSE, MAE, REL, and delta metrics. This work advances robust depth perception under dynamic conditions and paves the way for multi-modal, event-guided depth completion in real-world applications such as autonomous systems and robotics.

Abstract

Depth completion in dynamic scenes poses significant challenges due to rapid ego-motion and object motion, which can severely degrade the quality of input modalities such as RGB images and LiDAR measurements. Conventional RGB-D sensors often struggle to align precisely and capture reliable depth under such conditions. In contrast, event cameras with their high temporal resolution and sensitivity to motion at the pixel level provide complementary cues that are %particularly beneficial in dynamic environments.To this end, we propose EventDC, the first event-driven depth completion framework. It consists of two key components: Event-Modulated Alignment (EMA) and Local Depth Filtering (LDF). Both modules adaptively learn the two fundamental components of convolution operations: offsets and weights conditioned on motion-sensitive event streams. In the encoder, EMA leverages events to modulate the sampling positions of RGB-D features to achieve pixel redistribution for improved alignment and fusion. In the decoder, LDF refines depth estimations around moving objects by learning motion-aware masks from events. Additionally, EventDC incorporates two loss terms to further benefit global alignment and enhance local depth recovery. Moreover, we establish the first benchmark for event-based depth completion comprising one real-world and two synthetic datasets to facilitate future research. Extensive experiments on this benchmark demonstrate the superiority of our EventDC.

Figures (7)

• Figure 1: Data example and our solution for depth completion in dynamic environments. Leveraging high temporal resolution and motion sensitivity, event provides valuable complementary information for depth completion in dynamic scenes. Multiple event streams are aggregated for clear visualization.
• Figure 2: Pipeline of our EventDC. The color image $\mathbf{I}$, sparse depth $\mathbf{S}$, and event stream $\mathbf{E}$ are first processed by three structurally identical encoders. At each stage, the Event-Modulated Alignment (EMA) block leverages event features to align and fuse RGB-D representations. In the decoder, the Local Depth Filtering (LDF) unit further enhances depth estimation around moving objects, guided by the inherent sensitivity of events to motion and reinforced by local motion-aware constraints.
• Figure 3: Visualizations of the proposed EventDC benchmark: EventDC-Real/SemiSyn/FullSyn.
• Figure 4: Depth error comparisons on EventDC-Real. Warmer color indicates higher error.
• Figure 5: Visual results.