G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning

TL;DR

G^2VLM introduces a geometry-grounded Vision-Language Model that unifies 3D reconstruction and spatial understanding within a two-expert Mixture-of-Transformer-Experts framework. By training a geometric perception path alongside a semantic perception path, the model can directly predict 3D attributes from 2D inputs and leverage these priors for in-context spatial reasoning, achieving competitive 3D reconstruction results and state-of-the-art or near-state-of-the-art performance on multiple spatial benchmarks. A two-stage training regime enables scalable learning from abundant 2D multi-view data while incorporating 3D priors, with ablations showing a positive geometry–reasoning interplay. The approach provides a strong baseline for future tasks like 3D scene editing and demonstrates the value of integrating low-level geometry with high-level multimodal understanding.

Abstract

Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.

Figures (6)

• Figure 1: We present G$^2$VLM, a geometry grounded vision-language model proficient in both spatial 3D reconstruction and spatial understanding tasks. For spatial reasoning questions, G$^2$VLM can directly predict 3D geometry and employ interleaved reasoning for an answer.
• Figure 2: Our model, G$^2$VLM, employs an architecture inspired by the two-streams hypothesis. It features two experts: a geometric perception expert (our "where pathway") for visual geometry learning and a semantic perception expert (our "what pathway") for multimodal understanding.
• Figure 3: We present G$^2$VLM, a unified model that integrates both a geometric perception expert for 3D reconstruction and a semantic perception expert for multimodal understanding and spatial reasoning tasks. All tokens can do shared multi-modal self attention in each transformer block.
• Figure 4: Comparison of three different loss supervision mechanisms for the joint-training stage. Note that for visual geometry scores, lower is better. The VG + CE Loss approach yields the best performance, demonstrating that combining visual geometry and spatial understanding supervision mutually benefits spatial reasoning tasks.
• Figure 5: Qualitative results of our model. G$^2$VLM effectively reconstructs a diverse set of open-domain images, spanning object-level, structure-level, indoor, and outdoor scenes, including both dynamic and static content.