Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Driving

Qihao Sun, Jiarun Liu, Ziqian Ni, Jianyun Xu, Tao Xie, Lijun Zhao, Ruifeng Li, Sheng Yang

TL;DR

DriveMVS is presented, a novel multi-view stereo framework that reconciles these competing objectives through two key insights: Sparse but metrically accurate LiDAR observations can serve as geometric prompts to anchor depth estimation in absolute scale, and deep fusion of diverse cues is essential for resolving ambiguities and enhancing robustness.

Abstract

Accurate metric depth is critical for autonomous driving perception and simulation, yet current approaches struggle to achieve high metric accuracy, multi-view and temporal consistency, and cross-domain generalization. To address these challenges, we present DriveMVS, a novel multi-view stereo framework that reconciles these competing objectives through two key insights: (1) Sparse but metrically accurate LiDAR observations can serve as geometric prompts to anchor depth estimation in absolute scale, and (2) deep fusion of diverse cues is essential for resolving ambiguities and enhancing robustness, while a spatio-temporal decoder ensures consistency across frames. Built upon these principles, DriveMVS embeds the LiDAR prompt in two ways: as a hard geometric prior that anchors the cost volume, and as soft feature-wise guidance fused by a triple-cue combiner. Regarding temporal consistency, DriveMVS employs a spatio-temporal decoder that jointly leverages geometric cues from the MVS cost volume and temporal context from neighboring frames. Experiments show that DriveMVS achieves state-of-the-art performance on multiple benchmarks, excelling in metric accuracy, temporal stability, and zero-shot cross-domain transfer, demonstrating its practical value for scalable, reliable autonomous driving systems.

LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Driving

TL;DR

DriveMVS is presented, a novel multi-view stereo framework that reconciles these competing objectives through two key insights: Sparse but metrically accurate LiDAR observations can serve as geometric prompts to anchor depth estimation in absolute scale, and deep fusion of diverse cues is essential for resolving ambiguities and enhancing robustness.

Abstract

Accurate metric depth is critical for autonomous driving perception and simulation, yet current approaches struggle to achieve high metric accuracy, multi-view and temporal consistency, and cross-domain generalization. To address these challenges, we present DriveMVS, a novel multi-view stereo framework that reconciles these competing objectives through two key insights: (1) Sparse but metrically accurate LiDAR observations can serve as geometric prompts to anchor depth estimation in absolute scale, and (2) deep fusion of diverse cues is essential for resolving ambiguities and enhancing robustness, while a spatio-temporal decoder ensures consistency across frames. Built upon these principles, DriveMVS embeds the LiDAR prompt in two ways: as a hard geometric prior that anchors the cost volume, and as soft feature-wise guidance fused by a triple-cue combiner. Regarding temporal consistency, DriveMVS employs a spatio-temporal decoder that jointly leverages geometric cues from the MVS cost volume and temporal context from neighboring frames. Experiments show that DriveMVS achieves state-of-the-art performance on multiple benchmarks, excelling in metric accuracy, temporal stability, and zero-shot cross-domain transfer, demonstrating its practical value for scalable, reliable autonomous driving systems.
Paper Structure (54 sections, 19 equations, 12 figures, 8 tables)

This paper contains 54 sections, 19 equations, 12 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Monocular Depth Estimation Models.
  4. Multi-View Feed-Forward Models.
  5. Multi-View Stereo Models.
  6. Methodology
  7. Problem Definition
  8. Prompt-Anchored Cost Volume
  9. Triple-Cues Combiner
  10. Spatio-Temporal Decoder
  11. Implementation Details
  12. Prompt normalization.
  13. Training loss.
  14. Training data.
  15. Training strategy.
  16. ...and 39 more sections

Figures (12)

  • Figure 1: Overview of MVS-Pro. We introduce a Prompt-Anchored Cost Volume (\ref{['sec:prompt_anchored_cv']}) mechanism to fuse the absolute depth metric prompt into the multi-view cost volume. To fuse heterogeneous features from images, depth, and cost volume, we propose a Triple-Cues Combiner (\ref{['sec:triple_cues_combiner']}) to combine the cues. Finally, a Spatio-Temporal Decoder (\ref{['sec:spatio_temporal_decoder']}) produces continuous, consistent depth results.
  • Figure 2: The qualitative results of the estimated depth by different methods on KITTIgeiger2012kitti(row 1 & 2), DDADpacknet2020ddad(row 3) and Waymosun2020waymo(row 4 & 5). The best and second best are highlighted with green and yellow borders, respectively. Please check the red boxes in the figure for a detailed comparison.
  • Figure 3: Visualization of depth estimation on a static scene. The result demonstrates our robustness in challenging, low-parallax scenarios.
  • Figure 4: Ablations on (a) different LiDAR laser beams and (b) different LiDAR occlusion rate (from bottom). Lower AbsRel refers to a better result.
  • Figure 5: Ablation study on Lidar prompt absence. We estimate the depth of the back-view image while only the front-view image is provided by LiDAR. Our method remains an accurate metric under such a configuration.
  • ...and 7 more figures