IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction

TL;DR

IGGT proposes a unified instance-grounded framework that jointly learns 3D geometry and instance-level semantics from multi-view RGB inputs. A 1B-parameter large unified Transformer encodes the views into tokens decoded by a Geometry Head and an Instance Head, guided by a 3D-consistent contrastive loss $\mathcal{L}_{mvc}$ to enforce cross-view consistency. An Instance-Grounded Scene Understanding pipeline enables plug-and-play interaction with vision-language and multimodal models for open-vocabulary segmentation and QA grounding. InsScene-15K provides large-scale 3D-consistent instance annotations to support training and evaluation, and experiments on ScanNet/ScanNet++ demonstrate improved 3D coherence and downstream task performance compared to state-of-the-art methods.

Abstract

Humans naturally perceive the geometric structure and semantic content of a 3D world as intertwined dimensions, enabling coherent and accurate understanding of complex scenes. However, most prior approaches prioritize training large geometry models for low-level 3D reconstruction and treat high-level spatial understanding in isolation, overlooking the crucial interplay between these two fundamental aspects of 3D-scene analysis, thereby limiting generalization and leading to poor performance in downstream 3D understanding tasks. Recent attempts have mitigated this issue by simply aligning 3D models with specific language models, thus restricting perception to the aligned model's capacity and limiting adaptability to downstream tasks. In this paper, we propose InstanceGrounded Geometry Transformer (IGGT), an end-to-end large unified transformer to unify the knowledge for both spatial reconstruction and instance-level contextual understanding. Specifically, we design a 3D-Consistent Contrastive Learning strategy that guides IGGT to encode a unified representation with geometric structures and instance-grounded clustering through only 2D visual inputs. This representation supports consistent lifting of 2D visual inputs into a coherent 3D scene with explicitly distinct object instances. To facilitate this task, we further construct InsScene-15K, a large-scale dataset with high-quality RGB images, poses, depth maps, and 3D-consistent instance-level mask annotations with a novel data curation pipeline.

Figures (15)

• Figure 1: IGGT: building upon our curated large-scale dataset InsScene-15K, we propose a novel end-to-end framework that enables geometric reconstruction and contextual understanding in a unified representation. This paradigm facilitates a wide range of applications, including spatial tracking, 2D / 3D open-vocabulary segmentation, and scene grounding.
• Figure 2: Data Curation Pipeline. Our data is collected from various sources and then annotated by a novel data engine driven by SAM2 ravi2024sam2. (a) For video captured scenes (i.e., RE10k zhou2018stereo), we annotate them through a customized SAM2 video dense prediction pipeline. (b) For RGBD-scan scenes (e.g., ScanNet++ yeshwanth2023scannet++), we regenerate dense mask annotations for each image and align them with the projected coarse GT masks.
• Figure 3: Visualization of mask annotations from three different sources. For the RGBD-scan scene, we additionally compare the vanilla ground-truth masks from ScanNet++ yeshwanth2023scannet++ with our refined annotations, along with their corresponding matched IDs and mIoU scores.
• Figure 4: Overview of IGGT. Given input images, our method encodes them into a series of Unified Token Representations, which are then processed by the Geometry Head and the Instance Head to produce high-quality geometric reconstructions and instance-grounded clusterings simultaneously. In the end, we introduce Instance-Grounded Scene Understanding to perform multiple applications.
• Figure 5: Qualitative results on Instance Spatial Tracking. We present two example scenes from ScanNet dai2017scannet and ScanNet++ yeshwanth2023scannet++, and compare our method with SAM2* and SpaTracker+SAM. All instances are visualized with distinct IDs and colors for clarity.