DVGT: Driving Visual Geometry Transformer

TL;DR

DVGT addresses the challenge of obtaining dense, metric-scale 3D scene geometry for autonomous driving without relying on camera priors or post-alignment. It introduces a spatial-temporal transformer with factorized attention that processes unposed multi-view sequences to jointly predict a global 3D point map in ego coordinates and per-frame ego poses. Trained on a large mixture of driving datasets with a robust ground-truth construction pipeline, DVGT achieves state-of-the-art 3D reconstruction, depth, and ego-pose accuracy, directly in metric scale. This approach enhances flexibility across camera configurations and has meaningful implications for scalable, vision-centric driving systems.

Abstract

Perceiving and reconstructing 3D scene geometry from visual inputs is crucial for autonomous driving. However, there still lacks a driving-targeted dense geometry perception model that can adapt to different scenarios and camera configurations. To bridge this gap, we propose a Driving Visual Geometry Transformer (DVGT), which reconstructs a global dense 3D point map from a sequence of unposed multi-view visual inputs. We first extract visual features for each image using a DINO backbone, and employ alternating intra-view local attention, cross-view spatial attention, and cross-frame temporal attention to infer geometric relations across images. We then use multiple heads to decode a global point map in the ego coordinate of the first frame and the ego poses for each frame. Unlike conventional methods that rely on precise camera parameters, DVGT is free of explicit 3D geometric priors, enabling flexible processing of arbitrary camera configurations. DVGT directly predicts metric-scaled geometry from image sequences, eliminating the need for post-alignment with external sensors. Trained on a large mixture of driving datasets including nuScenes, OpenScene, Waymo, KITTI, and DDAD, DVGT significantly outperforms existing models on various scenarios. Code is available at https://github.com/wzzheng/DVGT.

Figures (8)

• Figure 1: DVGT is a large visual geometry transformer specifically designed for autonomous driving. It accepts a sequence of unposed multi-view images and predicts a metric-scaled global 3D point map in the ego-centric coordinate system and the ego poses for each frame, which outperforms the other geometry prediction models without post-alignment with external sensors.
• Figure 2:
• Figure 3: Framework of our DVGT for metric-scaled 3D scene geometry prediction. Given multi-frame, multi-view images, DVGT predicts metric-scaled 3D point map in the ego coordinate system of the first frame and the ego pose from each frame to the first frame.
• Figure 4: Comparison of our factorized spatial-temporal attention (left) against the global attention used in VGGT (right).
• Figure 5: We construct dense and accurate ground-truth 3D point maps for diverse driving scenarios. This figure showcases examples from the Waymo waymo, nuScenes nuscenes, OpenScene openscene, DDAD ddad, and KITTI kitti datasets. For each scene, we display the multi-view RGB images (left) and the corresponding high-quality 3D point map (right), highlighting the diversity and precision of our data.