Gen-LangSplat: Generalized Language Gaussian Splatting with Pre-Trained Feature Compression

TL;DR

Gen-LangSplat removes the per-scene language autoencoder bottleneck in open-vocabulary 3D grounding by introducing a generalized autoencoder trained on ScanNet to learn a transferable latent space for CLIP features. The pipeline uses SAM-derived masks and CLIP embeddings, compressed to a fixed $k$-dimensional latent space, and attaches learnable language features to each Gaussian in the 3D Splatting representation. After an RGB optimization, only the language codes are trained, enabling cross-scene language reasoning with about $2\times$ efficiency gains while preserving localization and segmentation performance comparable to LangSplat. An ablation study shows that a $16$-dimensional latent space optimally balances reconstruction fidelity (MSE ~$3\times10^{-4}$) and semantic retention (cosine similarity > $0.93$), supporting robust open-vocabulary 3D querying in novel environments.

Abstract

Modeling open-vocabulary language fields in 3D is essential for intuitive human-AI interaction and querying within physical environments. State-of-the-art approaches, such as LangSplat, leverage 3D Gaussian Splatting to efficiently construct these language fields, encoding features distilled from high-dimensional models like CLIP. However, this efficiency is currently offset by the requirement to train a scene-specific language autoencoder for feature compression, introducing a costly, per-scene optimization bottleneck that hinders deployment scalability. In this work, we introduce Gen-LangSplat, that eliminates this requirement by replacing the scene-wise autoencoder with a generalized autoencoder, pre-trained extensively on the large-scale ScanNet dataset. This architectural shift enables the use of a fixed, compact latent space for language features across any new scene without any scene-specific training. By removing this dependency, our entire language field construction process achieves a efficiency boost while delivering querying performance comparable to, or exceeding, the original LangSplat method. To validate our design choice, we perform a thorough ablation study empirically determining the optimal latent embedding dimension and quantifying representational fidelity using Mean Squared Error and cosine similarity between the original and reprojected 512-dimensional CLIP embeddings. Our results demonstrate that generalized embeddings can efficiently and accurately support open-vocabulary querying in novel 3D scenes, paving the way for scalable, real-time interactive 3D AI applications.

Figures (5)

• Figure 1: Visualization of learned 3D language features of the previous SOTA method, LangSplat, and our proposed approach. Without requiring per-scene training for the language-feature autoencoder, our method achieves comparable, or even superior, results while being more efficient.
• Figure 2: Overview of the proposed Gen-LangSplat framework. We leverage SAM to extract hierarchical semantics from multi-view images to resolve point ambiguity. The resulting segmentation masks are processed by the CLIP image encoder to obtain 512-D embeddings. These embeddings are compressed into a 16-D latent space using a generalized autoencoder pre-trained on ScanNet. Our 3D language Gaussians learn language features directly on a shared latent space derived from the pre-trained autoencoder. During querying, the rendered latent embeddings are decoded through the frozen decoder to recover the corresponding CLIP-space features for semantic reasoning.
• Figure 3: Qualitative comparison of open-vocabulary 3D object localization and segmentation results on the LERF dataset. The red points indicate model predictions, black dashed boxes denote the ground-truth annotations, and the bottom row shows the corresponding binary segmentation masks.
• Figure 4: Qualitative comparisons of 3D Segmentation on the 3D OVS Dataset.
• Figure 5: Ablation study on the latent dimensionality of our generalized autoencoder. As the dimension increases, the reconstruction error (MSE) decreases while the cosine similarity improves and saturates beyond $d=16$, indicating an optimal trade-off between compactness and semantic retention.