TSP3D: Text-guided Sparse Voxel Pruning for Efficient 3D Visual Grounding

TL;DR

TSP3D introduces a sparse, multi-level single-stage framework for real-time 3D visual grounding by coupling text guidance with voxel pruning and adaptive feature completion. Text-guided pruning ($TGP$) reduces voxel counts to enable cross-modal interaction, while completion-based addition ($CBA$) compensates for potential information loss, yielding robust grounding performance. The approach achieves state-of-the-art accuracy and unprecedented single-stage inference speeds on ScanRefer, NR3D, and SR3D, outperforming previous methods in both effectiveness and efficiency. This framework has practical impact for real-time robotics and AR/VR applications where rapid, accurate grounding from natural language is essential.

Abstract

In this paper, we propose an efficient multi-level convolution architecture for 3D visual grounding. Conventional methods are difficult to meet the requirements of real-time inference due to the two-stage or point-based architecture. Inspired by the success of multi-level fully sparse convolutional architecture in 3D object detection, we aim to build a new 3D visual grounding framework following this technical route. However, as in 3D visual grounding task the 3D scene representation should be deeply interacted with text features, sparse convolution-based architecture is inefficient for this interaction due to the large amount of voxel features. To this end, we propose text-guided pruning (TGP) and completion-based addition (CBA) to deeply fuse 3D scene representation and text features in an efficient way by gradual region pruning and target completion. Specifically, TGP iteratively sparsifies the 3D scene representation and thus efficiently interacts the voxel features with text features by cross-attention. To mitigate the affect of pruning on delicate geometric information, CBA adaptively fixes the over-pruned region by voxel completion with negligible computational overhead. Compared with previous single-stage methods, our method achieves top inference speed and surpasses previous fastest method by 100\% FPS. Our method also achieves state-of-the-art accuracy even compared with two-stage methods, with $+1.13$ lead of Acc@0.5 on ScanRefer, and $+2.6$ and $+3.2$ leads on NR3D and SR3D respectively. The code is available at \href{https://github.com/GWxuan/TSP3D}{https://github.com/GWxuan/TSP3D}.

Figures (7)

• Figure 1: Comparison of 3DVG methods on ScanRefer dataset chen2020scanrefer. Our TSP3D surpasses existing methods in both accuracy and inference speed, achieving the first efficient 3DVG framework.
• Figure 2: Illustration of TSP3D. TSP3D bulids on multi-level sparse convolutional architecture. It iteratively upsamples the voxel features with text-guided pruning (TGP), and fuses multi-level features via completion-based addition (CBA). (a) to (d) on the right side illustrate various options for feature upsampling. (a) refers to simple concatenation with text features, which is fast but less accurate. (b) refers to feature interaction through cross-modal attention mechanisms, which is constrained by the large number of voxels. (c) represents our proposed TGP, which first prunes voxel features under textual guidance and thus enables efficient interaction between voxel and text features. (d) shows a simplified version of TGP that removes farthest point sampling and interpolation, combines multi-modal feature interactions into a whole and moves it before pruning.
• Figure 3: Illustration of completion-based addition. The upper figure (b) illustrates an example of over-pruning on the target. The lower figure (c) shows the completed features predicted by CBA.
• Figure 4: Visualization of the text-guided pruning process. In each example, the voxel features after scene-level TGP, target-level TGP and the last upsampling layer are presented from top to bottom. The blue boxes represent the ground truth of the target, and the red boxes denote the bounding boxes of relevant objects. TSP3D reduces the amount of voxel features through two stages of pruning and progressively guides the network focusing towards the target.
• Figure 5: Visualization of the completion-based addition process. The blue points represent the voxel features output by the target-level TGP, while the red points are the completion features predicted by the CBA. The blue boxes indicate the ground truth boxes. CBA adaptively supplements situations where excessive pruning has occurred.