VGD: Visual Geometry Gaussian Splatting for Feed-Forward Surround-view Driving Reconstruction

TL;DR

This work tackles fast, generalizable surround-view reconstruction for autonomous driving by decomposing the problem into geometry extraction, Gaussian-based view synthesis, and semantic refinement. The core idea is to explicitly learn geometry and propagate it into a Gaussian Splatting renderer, guided by a distilled VGGT priors through a lightweight, patch-based DVGT-like backbone, followed by a DPT-GS Gaussian head and a multi-scale refinement module. The three components are trained end-to-end with a composite loss that combines distillation, render quality, and scale-aware localization, enabling coherent geometry and semantics across sparse views. On nuScenes, VGD achieves state-of-the-art performance in both single-frame and multi-frame settings, offering substantial improvements in visual fidelity while maintaining fast, generalizable inference suitable for real-time autonomous driving pipelines.

Abstract

Feed-forward surround-view autonomous driving scene reconstruction offers fast, generalizable inference ability, which faces the core challenge of ensuring generalization while elevating novel view quality. Due to the surround-view with minimal overlap regions, existing methods typically fail to ensure geometric consistency and reconstruction quality for novel views. To tackle this tension, we claim that geometric information must be learned explicitly, and the resulting features should be leveraged to guide the elevating of semantic quality in novel views. In this paper, we introduce \textbf{Visual Gaussian Driving (VGD)}, a novel feed-forward end-to-end learning framework designed to address this challenge. To achieve generalizable geometric estimation, we design a lightweight variant of the VGGT architecture to efficiently distill its geometric priors from the pre-trained VGGT to the geometry branch. Furthermore, we design a Gaussian Head that fuses multi-scale geometry tokens to predict Gaussian parameters for novel view rendering, which shares the same patch backbone as the geometry branch. Finally, we integrate multi-scale features from both geometry and Gaussian head branches to jointly supervise a semantic refinement model, optimizing rendering quality through feature-consistent learning. Experiments on nuScenes demonstrate that our approach significantly outperforms state-of-the-art methods in both objective metrics and subjective quality under various settings, which validates VGD's scalability and high-fidelity surround-view reconstruction.

Figures (7)

• Figure 1: Comparison of feed-forward reconstruction paradigms. Previous methods 12_tian2025drivingforward (top) suffer from poor rendering quality due to unstable 2D-to-3D lifting geometry for direct rendering. In contrast, our VGD (bottom) distills strong geometric priors from a pre-trained model and jointly optimizes semantic information, ensuring geometric stability and high-fidelity rendering.
• Figure 2: The end-to-end architecture of VGD. Our framework transforms surround-view inputs into novel views in three stages. First, a DPT-Depth branch, supervised by a pre-trained VGGT, extracts strong geometric priors. Second, a DPT-GS head uses these geometric features to guide the prediction of other Gaussian attributes. Finally, a fusion model refines the initial renderings using multi-scale features from both branches to produce the high-fidelity output.
• Figure 3: Qualitative results of novel surround-view on Single-Frame Mode. Red rectangular boxes highlight the differences.
• Figure 4: Qualitative results of novel surround-view on Multi-Frame Mode. Red rectangular boxes highlight the differences.
• Figure 5: Qualitative results of extreme case on Single-Frame Mode. Red rectangular boxes highlight the differences.