LiDAR-VGGT: Cross-Modal Coarse-to-Fine Fusion for Globally Consistent and Metric-Scale Dense Mapping

TL;DR

LiDAR-VGGT addresses the lack of metric scale in VGGT and calibration sensitivity in LIVO by fusing LiDAR with VGGT in a two-stage pipeline. A coarse Pre-Fusion Module refines VGGT poses using LiDAR-based scale through scale-aware RANSAC and linearity validation, establishing an approximate metric scale, while a Fine Post-Fusion Module applies regularized cross-modal $Sim(3)$ registration with bounding-box constraints and a global pose graph optimization to ensure metric-scale, globally consistent reconstructions. A four-metric colored point-cloud evaluation toolkit is introduced and validated across diverse datasets, showing superior geometric fidelity and color quality compared to VGGT-based methods and LIVO baselines. The results demonstrate dense, metrically accurate maps that are robust to extrinsic calibration and synchronization issues, with future work aiming to integrate LiDAR cues directly into the VGGT backbone for end-to-end cross-modal reconstruction.

Abstract

Reconstructing large-scale colored point clouds is an important task in robotics, supporting perception, navigation, and scene understanding. Despite advances in LiDAR inertial visual odometry (LIVO), its performance remains highly sensitive to extrinsic calibration. Meanwhile, 3D vision foundation models, such as VGGT, suffer from limited scalability in large environments and inherently lack metric scale. To overcome these limitations, we propose LiDAR-VGGT, a novel framework that tightly couples LiDAR inertial odometry with the state-of-the-art VGGT model through a two-stage coarse- to-fine fusion pipeline: First, a pre-fusion module with robust initialization refinement efficiently estimates VGGT poses and point clouds with coarse metric scale within each session. Then, a post-fusion module enhances cross-modal 3D similarity transformation, using bounding-box-based regularization to reduce scale distortions caused by inconsistent FOVs between LiDAR and camera sensors. Extensive experiments across multiple datasets demonstrate that LiDAR-VGGT achieves dense, globally consistent colored point clouds and outperforms both VGGT-based methods and LIVO baselines. The implementation of our proposed novel color point cloud evaluation toolkit will be released as open source.

Figures (9)

• Figure 1: Motivation figure: (a) and (b) present point clouds from FAST-LIO2 fastlio2 and FAST-LIVO2 fastlivo2, which are sparse and contain holes due to limited LiDAR scanning coverage. (c) shows the dense yet scale-ambiguous map produced by VGGT vggt. (d) illustrates our method, which generates VGGT-based point clouds guided by LiDAR reconstruction, achieving dense maps with accurate metric scale.
• Figure 2: Overview of the proposed LiDAR-VGGT system. It takes multi-modal data from IMU, LiDAR, and camera as input and produces globally consistent, dense reconstructions with metric scale. Our pipeline follows a coarse-to-fine strategy, consisting of a pre-fusion module (Sec. \ref{['subsetcion:frontend']}) and a post-fusion module (Sec. \ref{['subsection:backend']}). The former fuses LIO and VGGT poses to establish an approximate real-world scale for VGGT-generated scene attributes, while the latter refines the VGGT-colored point cloud through enhanced cross-modal $Sim(3)$ registration and global pose graph optimization.
• Figure 3: (a) demonstrates an example with low linearity of LIO poses and thus yielding incorrect scale after pose registration. (b) shows the result with our scale correction which recovers an accurate scale close to the real value.
• Figure 4: Structure of our global PGO. The first frame of the first session is fixed as the world reference frame.
• Figure 5: Our two data collection platforms. The UAV device (a) is used for outdoor data collection, while the handheld device (b) is used for indoor.