GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splatting

TL;DR

GAGS tackles open-vocabulary 3D grounding by distilling CLIP-derived 2D features into a single 3D Gaussian field built on 3D Gaussian Splatting. It introduces granularity-aware segmentation and a learnable granularity decoding mechanism to ensure multiview-consistent features, addressing the core challenge of 2D-to-3D feature inconsistency. The method achieves notable gains in both localization and semantic segmentation on LERF and Mip-NeRF-360, while delivering roughly double the inference speed compared to prior multi-granularity approaches. Overall, GAGS provides a robust, efficient framework for open-vocabulary 3D scene understanding with practical applicability to interactive 3D querying and rendering.

Abstract

3D open-vocabulary scene understanding, which accurately perceives complex semantic properties of objects in space, has gained significant attention in recent years. In this paper, we propose GAGS, a framework that distills 2D CLIP features into 3D Gaussian splatting, enabling open-vocabulary queries for renderings on arbitrary viewpoints. The main challenge of distilling 2D features for 3D fields lies in the multiview inconsistency of extracted 2D features, which provides unstable supervision for the 3D feature field. GAGS addresses this challenge with two novel strategies. First, GAGS associates the prompt point density of SAM with the camera distances, which significantly improves the multiview consistency of segmentation results. Second, GAGS further decodes a granularity factor to guide the distillation process and this granularity factor can be learned in a unsupervised manner to only select the multiview consistent 2D features in the distillation process. Experimental results on two datasets demonstrate significant performance and stability improvements of GAGS in visual grounding and semantic segmentation, with an inference speed 2$\times$ faster than baseline methods. The code and additional results are available at https://pz0826.github.io/GAGS-Webpage/ .

Figures (16)

• Figure 1: Given multiview images of a scene, GAGS learns a 3D Gaussian field associated with semantic features, which enables accurate open-vocabulary 3D visual grounding in the scene.
• Figure 2: A 3D point shows different semantics at different scales.
• Figure 3: In comparison to vanilla SAM, we associate the prompt point density with distance to the object, which greatly enhances the consistency of multi-view segmentation.
• Figure 4: Our granularity-select feature learning. Our method leverages the inherent consistency of 3D Gaussian splatting to perform granularity-aware feature distillation, enhancing the stability and accuracy of learned object features.
• Figure 5: Pipeline of our method. Given a set of images with camera poses, our method first uses 3D Gaussian Splatting to reconstruct the scene's geometric representation, then utilizes it for granularity-aware segmentation and CLIP feature distillation. The finally output 3D feature field supports open-vocabulary queries.