OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding

TL;DR

OpenVoxel tackles open-vocabulary 3D scene understanding without training embeddings by constructing a scene map from SVR voxels via per-view SAM2 masks, canonical captions, and MLLMs. It avoids training language embeddings and instead uses text-to-text reasoning over a rich, captioned scene map to perform OVS and RES. The method delivers strong RES performance while remaining fast and annotation-free, thanks to its training-free voxel grouping and retrieval pipeline. This approach offers a scalable path toward flexible, real-world 3D understanding without costly annotation or embedding training.

Abstract

We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene understanding tasks. Given the sparse voxel rasterization (SVR) model obtained from multi-view images of a 3D scene, our OpenVoxel is able to produce meaningful groups that describe different objects in the scene. Also, by leveraging powerful Vision Language Models (VLMs) and Multi-modal Large Language Models (MLLMs), our OpenVoxel successfully build an informative scene map by captioning each group, enabling further 3D scene understanding tasks such as open-vocabulary segmentation (OVS) or referring expression segmentation (RES). Unlike previous methods, our method is training-free and does not introduce embeddings from a CLIP/BERT text encoder. Instead, we directly proceed with text-to-text search using MLLMs. Through extensive experiments, our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks. The code will be open.

Figures (10)

• Figure 1: Comparison of lifting language into 3D representations between ReferSplat refersplat and OpenVoxel. Note that ReferSplat additionally requires manual text annotations for training 3DGS to equipped with Bert embeddings, while ours is totally training-free and do not need any additional annotations.
• Figure 2: Overview of OpenVoxel. Taking a sparse voxel model $V_{1:N}$ pre-trained from multi-view images $I_{1:K}$ and their corresponding camera pose $\xi_{1:K}$, we aim to build a voxel group field $\mathcal{F}_{1:N}$ from segmentation masks $M_{1:K}$ obtained from SAM2. With $\mathcal{F}_{1:N}$ we render images and masks for all groups to obtain canonical captions and construct a Scene Map $S$ recoding their position and captions. In the Referring Query Inference stage we take a query description and image to find the ideal target group that matches the description by query refinement and text-to-text retrieving from $S$, enabling complex segmentation tasks such as referring expression segmentation (RES).
• Figure 3: Detail of the grouping process. Taking the pre-trained voxel model $V_{1:K}$, we initialize the Group Field $\mathcal{F}^0_{1:N}$, Feature weight $W_{1:N}^0$ as empty tensors, and Group Dictionary $G^0$ as empty dictionary. Then start from $\xi_{1}$, we project the SAM masks $M_1$ to 3D voxel and update $\mathcal{F}_{1:N}$, $W_{1:N}$, and $G$. By match and merge masks from the other views repeating this process, the final $\mathcal{F}_{1:N}$, $W_{1:N}$, and $G$ is able to represent the group information of $V_{1:N}$.
• Figure 4: Detail of the Canonical Captioning. Given the group masks rendered of a specific group (taking the green apple as example) from our group field and their corresponding images, we leverage the Describe Anything Model (DAM) to first obtain a detailed caption. Then a Qwen3-VL model is conducted to canonicalize the caption into a fixed form, benefiting further usage.
• Figure 5: Qualitative results of RES task on the Figurines scene.