Driv3R: Learning Dense 4D Reconstruction for Autonomous Driving

TL;DR

Driv3R tackles real-time dense 4D reconstruction for dynamic driving scenes by regressing per-frame 4D point maps from multi-view images into a global world frame without optimization. It introduces a temporal-spatial memory pool to fuse temporal and cross-view information, coupled with a 4D flow predictor to focus on moving objects, and an optimization-free multi-view aligner to ensure world-coordinate consistency. With depth supervision from a pretrained R3D3 model, Driv3R achieves strong 4D reconstruction performance and up to 15x faster inference than global-alignment baselines on nuScenes. The approach offers a scalable, streaming solution for autonomous driving perception, significantly reducing compute while maintaining accuracy for dynamic scenes.

Abstract

Realtime 4D reconstruction for dynamic scenes remains a crucial challenge for autonomous driving perception. Most existing methods rely on depth estimation through self-supervision or multi-modality sensor fusion. In this paper, we propose Driv3R, a DUSt3R-based framework that directly regresses per-frame point maps from multi-view image sequences. To achieve streaming dense reconstruction, we maintain a memory pool to reason both spatial relationships across sensors and dynamic temporal contexts to enhance multi-view 3D consistency and temporal integration. Furthermore, we employ a 4D flow predictor to identify moving objects within the scene to direct our network focus more on reconstructing these dynamic regions. Finally, we align all per-frame pointmaps consistently to the world coordinate system in an optimization-free manner. We conduct extensive experiments on the large-scale nuScenes dataset to evaluate the effectiveness of our method. Driv3R outperforms previous frameworks in 4D dynamic scene reconstruction, achieving 15x faster inference speed compared to methods requiring global alignment. Code: https://github.com/Barrybarry-Smith/Driv3R.

Figures (7)

• Figure 1: Our Driv3R predicts dense 4D dynamic point clouds in the global world coordinate system in a streaming manner from multi-view images. It outperforms existing methods in reconstructing dynamic autonomous driving scenes and achieves a $15 \times$ faster inference speed compared to approaches that require global alignment optimization zhang2024monst3rdust3r_cvpr24.
• Figure 2: Temporal and spatial interactions within the memory pool. By maintaining a sensor-aware memory pool where key-value pairs are stored in the order of timestamps, we clearly identify both spatial and temporal relationships and perform a more efficient feature update process.
• Figure 3: Overview of Driv3R. Given multi-view input sequences, we construct a sensor-wise memory pool for temporal and spatial information interactions. After obtaining per-frame point maps, the 4D flow predictor identifies the dynamic objects within the scene. Finally, we adopt an optimization-free multi-view alignment strategy to predict the 4D global point cloud in the world coordinate system.
• Figure 4: Visualization Results of Depth Prediction on nuScenes nuscenes2019 dataset. In these cases, Driv3R leverages the R3D3 r3d3 model to mitigate the blur in Spann3R wang20243d caused by fast motion, resulting in more precise reconstruction of the dynamic scenes.
• Figure 5: Visualization Results of 4D Reconstruction on nuScenes nuscenes2019. Driv3R leverages both strong 3D priors from DUSt3R dust3r_cvpr24 and accurate dynamic predictions in R3D3 r3d3, while maintaining both temporal and spatial consistency on 4D reconstruction.