Quantile Rendering: Efficiently Embedding High-dimensional Feature on 3D Gaussian Splatting

TL;DR

This work tackles the challenge of embedding high-dimensional features for open-vocabulary segmentation within 3D Gaussian Splatting. It introduces Quantile Rendering (Q-Render), a transmittance-guided, sparse sampling strategy that selects a small set of dominant Gaussians per ray to render 512-D features efficiently, and couples it with a generalizable 3D neural network (GS-Net) that predicts per-Gaussian features from optimized 3D Gaussians. The approach yields state-of-the-art results on ScanNet and LeRF-OVS, while achieving up to ~43.7x speedups for high-dimensional feature rendering, enabling real-time open-vocabulary 3D segmentation. These contributions bridge 2D foundation-model embeddings with 3D Gaussian representations, delivering scalable, high-fidelity open-vocabulary understanding in indoor and outdoor-like scenes, though some limitations remain in adaptive sampling and reliance on per-scene 3D-GS pipelines.

Abstract

Recent advancements in computer vision have successfully extended Open-vocabulary segmentation (OVS) to the 3D domain by leveraging 3D Gaussian Splatting (3D-GS). Despite this progress, efficiently rendering the high-dimensional features required for open-vocabulary queries poses a significant challenge. Existing methods employ codebooks or feature compression, causing information loss, thereby degrading segmentation quality. To address this limitation, we introduce Quantile Rendering (Q-Render), a novel rendering strategy for 3D Gaussians that efficiently handles high-dimensional features while maintaining high fidelity. Unlike conventional volume rendering, which densely samples all 3D Gaussians intersecting each ray, Q-Render sparsely samples only those with dominant influence along the ray. By integrating Q-Render into a generalizable 3D neural network, we also propose Gaussian Splatting Network (GS-Net), which predicts Gaussian features in a generalizable manner. Extensive experiments on ScanNet and LeRF demonstrate that our framework outperforms state-of-the-art methods, while enabling real-time rendering with an approximate ~43.7x speedup on 512-D feature maps. Code will be made publicly available.

Figures (14)

• Figure 1: Quantile rendering. (a) Unlike volume rendering nerf3dgs that densely samples and blends all 3D Gaussians along the rays, (b) our Quantile render selectively samples and blends a sparse set of quantile Gaussians -- those with dominant influence along the ray, which can efficiently render high-dimensional feature maps from Gaussian features.
• Figure 2: Overview. Given optimized 3D Gaussians $\mathcal{G}$, our network is trained to predict Gaussian features $\mathcal{F}$ that are aligned with the language embedding space from CLIP's vision encoder. Typically, the proposed Q-Render accelerates the training and inference speed by transforming predicted Gaussian features into rendered feature maps.
• Figure 3: Comparison of transmittance distribution across different Gaussians sampling algorithms. Our Q-Render effectively approximates the distribution of the transmittance distribution of the original 3D-GS. We used $K=10$ for visualization.
• Figure 4: Qualitative results on the open-vocabulary 3D semantic segmentation task.
• Figure 5: Our qualitative results in the LeRF-OVS dataset.