Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VG3T: Visual Geometry Grounded Gaussian Transformer

Junho Kim, Seongwon Lee

TL;DR

VG3T introduces an end-to-end multi-view framework that directly regresses 3D Gaussian primitives from surround-view images, enabling early cross-view fusion to produce coherent 3D semantic occupancy. A two-stage density-bias mitigation pipeline—Grid-Based Sampling and Positional Refinement—reduces redundant primitives near cameras and enhances detail in sparse regions. The method achieves state-of-the-art nuScenes results with fewer Gaussians than prior Gaussian-based approaches, demonstrating improved efficiency and accuracy in 3D scene understanding. Overall, VG3T advances sparse, semantically informed 3D representations for autonomous driving by unifying geometry and semantics in a single, trainable pipeline.

Abstract

Generating a coherent 3D scene representation from multi-view images is a fundamental yet challenging task. Existing methods often struggle with multi-view fusion, leading to fragmented 3D representations and sub-optimal performance. To address this, we introduce VG3T, a novel multi-view feed-forward network that predicts a 3D semantic occupancy via a 3D Gaussian representation. Unlike prior methods that infer Gaussians from single-view images, our model directly predicts a set of semantically attributed Gaussians in a joint, multi-view fashion. This novel approach overcomes the fragmentation and inconsistency inherent in view-by-view processing, offering a unified paradigm to represent both geometry and semantics. We also introduce two key components, Grid-Based Sampling and Positional Refinement, to mitigate the distance-dependent density bias common in pixel-aligned Gaussian initialization methods. Our VG3T shows a notable 1.7%p improvement in mIoU while using 46% fewer primitives than the previous state-of-the-art on the nuScenes benchmark, highlighting its superior efficiency and performance.

VG3T: Visual Geometry Grounded Gaussian Transformer

TL;DR

VG3T introduces an end-to-end multi-view framework that directly regresses 3D Gaussian primitives from surround-view images, enabling early cross-view fusion to produce coherent 3D semantic occupancy. A two-stage density-bias mitigation pipeline—Grid-Based Sampling and Positional Refinement—reduces redundant primitives near cameras and enhances detail in sparse regions. The method achieves state-of-the-art nuScenes results with fewer Gaussians than prior Gaussian-based approaches, demonstrating improved efficiency and accuracy in 3D scene understanding. Overall, VG3T advances sparse, semantically informed 3D representations for autonomous driving by unifying geometry and semantics in a single, trainable pipeline.

Abstract

Generating a coherent 3D scene representation from multi-view images is a fundamental yet challenging task. Existing methods often struggle with multi-view fusion, leading to fragmented 3D representations and sub-optimal performance. To address this, we introduce VG3T, a novel multi-view feed-forward network that predicts a 3D semantic occupancy via a 3D Gaussian representation. Unlike prior methods that infer Gaussians from single-view images, our model directly predicts a set of semantically attributed Gaussians in a joint, multi-view fashion. This novel approach overcomes the fragmentation and inconsistency inherent in view-by-view processing, offering a unified paradigm to represent both geometry and semantics. We also introduce two key components, Grid-Based Sampling and Positional Refinement, to mitigate the distance-dependent density bias common in pixel-aligned Gaussian initialization methods. Our VG3T shows a notable 1.7%p improvement in mIoU while using 46% fewer primitives than the previous state-of-the-art on the nuScenes benchmark, highlighting its superior efficiency and performance.

Paper Structure

This paper contains 20 sections, 14 equations, 4 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. 3D Semantic Occupancy Prediction
  4. Multi-view 3D Reconstruction
  5. Visual Geometry Grounded Gaussian Transformer (VG3T)
  6. Problem Setup and Overview
  7. Multi-View Transformer Backbone
  8. Gaussian Initialization
  9. Grid-Based Sampling
  10. Grid-Based Sampling
  11. Positional Refinement
  12. 3D Occupancy Rendering
  13. Training Objective
  14. Experiments
  15. Implementation Details
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Coherent 3D Scene Representation via Early Multi-View Fusion. Unlike prior work that processes each camera view independently, leading to fragmented and inconsistent 3D representations, our method, VG3T, leverages early cross-view correlation. This paradigm produces a coherent and geometrically accurate 3D representation.
  • Figure 2: Overview of the Visual Geometry Grounded Gaussian Transformer (VG3T) Architecture.
  • Figure 3: Qualitative results of our method and GaussianFormer-2 on SurroundOcc dataset.
  • Figure 4: Qualitative results of model component design.