DELTA: Dense Depth from Events and LiDAR using Transformer's Attention

TL;DR

DELTA addresses dense depth estimation by fusing asynchronous event data with LiDAR through a transformer-based attention framework. It introduces a propagation memory and a central memory to enable high-rate, temporally coherent fusion, and uses self- and cross-attention to model intra- and inter-modal relationships, producing a single depth map per temporal window $D_{bf}$. Across SLED, MVSEC, and M3ED, DELTA achieves state-of-the-art or competitive results, with particularly strong improvements at close range (up to a fourfold error reduction) and robust performance under challenging lighting and motion. The work lays a foundation for high-rate, multi-sensor depth densification in automotive perception and suggests directions for further optimization, including alternative event representations and 3D fusion extensions.

Abstract

Event cameras and LiDARs provide complementary yet distinct data: respectively, asynchronous detections of changes in lighting versus sparse but accurate depth information at a fixed rate. To this day, few works have explored the combination of these two modalities. In this article, we propose a novel neural-network-based method for fusing event and LiDAR data in order to estimate dense depth maps. Our architecture, DELTA, exploits the concepts of self- and cross-attention to model the spatial and temporal relations within and between the event and LiDAR data. Following a thorough evaluation, we demonstrate that DELTA sets a new state of the art in the event-based depth estimation problem, and that it is able to reduce the errors up to four times for close ranges compared to the previous SOTA.

Figures (23)

• Figure 1: Overall principle of DELTA. Sparse and low-rate projected LiDAR data () is densified spatially and temporally using higher-rate temporal windows of event data (), resulting in dense high-rate depth maps (). Displayed here is an example of the high-quality depth maps produced by DELTA.
• Figure 2: The complete architecture of our DELTA network. Unless noted, data is of shape $(N, D)$, where $N$ is the number of patches, and $D$ their dimensionality (please refer to \ref{['sec:method:encod_heads', 'sec:eval:impl_detail:data_size']} for more details).
• Figure 3: Results on the ((a), (b)) and ((c), (d)) sequences of SLED. Rows (a) and (c), left to right: ground truth depth map; result from ALEDSL; our result (DELTASL). Differences between ALEDSL and DELTASL are better seen in rows (b) and (d), showing the error maps of ALEDSL and DELTASL (where pixels with an error inferior to 0.5m are in gray). For a better visualization, an enlarged version of this figure is given in the Supplementary Material.
• Figure 4: Qualitative results on the MVSEC dataset. Left to right: events; LiDAR projection (with larger points for a better readability); ground truth; results from Cui et al. Cui2022DenseDE; results from ALED Brebion2023LearningTE; our results. Top to bottom: ; ; . For a better visualization, an enlarged version of this figure is given in the Supplementary Material.
• Figure 5: Qualitative results for the sequence from M3ED. The size of points was increased for both the LiDAR projection and the ground truth. For a better visualization, an enlarged version of this figure is given in the Supplementary Material.