Language and Geometry Grounded Sparse Voxel Representations for Holistic Scene Understanding

TL;DR

This work proposes a novel approach that leverages language and geometry grounded sparse voxel representations to comprehensively model appearance, semantics, and geometry within a unified framework and achieves superior overall performance compared with state-of-the-art methods in holistic scene understanding and reconstruction.

Abstract

Existing 3D open-vocabulary scene understanding methods mostly emphasize distilling language features from 2D foundation models into 3D feature fields, but largely overlook the synergy among scene appearance, semantics, and geometry. As a result, scene understanding often deviates from the underlying geometric structure of scenes and becomes decoupled from the reconstruction process. In this work, we propose a novel approach that leverages language and geometry grounded sparse voxel representations to comprehensively model appearance, semantics, and geometry within a unified framework. Specifically, we use 3D sparse voxels as primitives and employ an appearance field, a density field, a feature field, and a confidence field to holistically represent a 3D scene. To promote synergy among the appearance, density, and feature fields, we construct a feature modulation module and distill language features from a 2D foundation model into our 3D scene model. In addition, we integrate geometric distillation into feature field distillation to transfer geometric knowledge from a geometry foundation model to our 3D scene representations via depth correlation regularization and pattern consistency regularization. These components work together to synergistically model the appearance, semantics, and geometry of the 3D scene within a unified framework. Extensive experiments demonstrate that our approach achieves superior overall performance compared with state-of-the-art methods in holistic scene understanding and reconstruction.

Figures (9)

• Figure 1: An illustration of our approach to holistic scene understanding. Our approach leverages language and geometry grounded sparse voxel representations to comprehensively model the appearance, semantics, and geometry of the 3D scene in a unified framework, achieving better overall performance compared with the state-of-the-arts.
• Figure 2: An overview of the proposed approach. Our approach optimizes language and geometry grounded sparse voxels as the 3D scene representations to comprehensively model the appearance, semantics, and geometry of the 3D scene in a unified framework.
• Figure 3: The illustrations of (a) the proposed feature modulation module and (b) the architecture of the employed autoencoder. We show the hidden dimension of each layer above the corresponding layer.
• Figure 4: Qualitative comparison with state-of-the-art methods in 3D semantic segmentation on the LERF and Mip-NeRF360 datasets. The segmented regions are highlighted.
• Figure 5: Qualitative comparison with state-of-the-art methods in 3D object localization on the LERF and Mip-NeRF360 datasets. Red dots denote the positions with the highest localization responses and black dashed boxes represent the ground-truth localizations.