Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastLGS: Speeding up Language Embedded Gaussians with Feature Grid Mapping

Yuzhou Ji, He Zhu, Junshu Tang, Wuyi Liu, Zhizhong Zhang, Xin Tan, Yuan Xie

TL;DR

FastLGS tackles open-vocabulary semantic querying in high-resolution 3D Gaussian Splatting by introducing semantic feature grids that map multi-view CLIP features (derived from SAM masks) into low-dimensional representations tied to Gaussians. The method employs cross-view grid mapping to achieve object-centric, view-consistent semantics and enables pixel-aligned, language-grounded queries with real-time performance. It demonstrates state-of-the-art speed and accuracy on open-vocabulary retrieval, while supporting downstream tasks such as language-driven 3D segmentation and object inpainting. This work enables practical, interactive 3D semantic understanding and manipulation in real-world scenarios.

Abstract

The semantically interactive radiance field has always been an appealing task for its potential to facilitate user-friendly and automated real-world 3D scene understanding applications. However, it is a challenging task to achieve high quality, efficiency and zero-shot ability at the same time with semantics in radiance fields. In this work, we present FastLGS, an approach that supports real-time open-vocabulary query within 3D Gaussian Splatting (3DGS) under high resolution. We propose the semantic feature grid to save multi-view CLIP features which are extracted based on Segment Anything Model (SAM) masks, and map the grids to low dimensional features for semantic field training through 3DGS. Once trained, we can restore pixel-aligned CLIP embeddings through feature grids from rendered features for open-vocabulary queries. Comparisons with other state-of-the-art methods prove that FastLGS can achieve the first place performance concerning both speed and accuracy, where FastLGS is 98x faster than LERF and 4x faster than LangSplat. Meanwhile, experiments show that FastLGS is adaptive and compatible with many downstream tasks, such as 3D segmentation and 3D object inpainting, which can be easily applied to other 3D manipulation systems.

FastLGS: Speeding up Language Embedded Gaussians with Feature Grid Mapping

TL;DR

FastLGS tackles open-vocabulary semantic querying in high-resolution 3D Gaussian Splatting by introducing semantic feature grids that map multi-view CLIP features (derived from SAM masks) into low-dimensional representations tied to Gaussians. The method employs cross-view grid mapping to achieve object-centric, view-consistent semantics and enables pixel-aligned, language-grounded queries with real-time performance. It demonstrates state-of-the-art speed and accuracy on open-vocabulary retrieval, while supporting downstream tasks such as language-driven 3D segmentation and object inpainting. This work enables practical, interactive 3D semantic understanding and manipulation in real-world scenarios.

Abstract

The semantically interactive radiance field has always been an appealing task for its potential to facilitate user-friendly and automated real-world 3D scene understanding applications. However, it is a challenging task to achieve high quality, efficiency and zero-shot ability at the same time with semantics in radiance fields. In this work, we present FastLGS, an approach that supports real-time open-vocabulary query within 3D Gaussian Splatting (3DGS) under high resolution. We propose the semantic feature grid to save multi-view CLIP features which are extracted based on Segment Anything Model (SAM) masks, and map the grids to low dimensional features for semantic field training through 3DGS. Once trained, we can restore pixel-aligned CLIP embeddings through feature grids from rendered features for open-vocabulary queries. Comparisons with other state-of-the-art methods prove that FastLGS can achieve the first place performance concerning both speed and accuracy, where FastLGS is 98x faster than LERF and 4x faster than LangSplat. Meanwhile, experiments show that FastLGS is adaptive and compatible with many downstream tasks, such as 3D segmentation and 3D object inpainting, which can be easily applied to other 3D manipulation systems.
Paper Structure (18 sections, 5 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 18 sections, 5 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. NeRF and 3D Gaussian Splatting (3DGS)
  4. 2D and 3D Segmentation
  5. Semantic 3D scene
  6. Method
  7. Overview
  8. Initialization
  9. Semantic Feature Grid
  10. Cross View Grid Mapping
  11. Training Features for Gaussians
  12. Querying Features
  13. Experiments
  14. Comparison With Other State-of-the-art Methods
  15. Downstream Applications
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Visualized relevancy (turbo) of query "Xbox wireless controller". Our result is much more accurate in having higher relevancy within queried areas and lower relevancy among other regions compared with LangSplat.
  • Figure 2: FastLGS pipeline. Left: Initialization. Mid: Feature grid construction and embeddings restoring. Right: query using open-vocabulary prompts.
  • Figure 3: Visual results of retrieved objects in different scenes.
  • Figure 4: Ablation on keypoint and color feature matching.