3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding

TL;DR

3DRP-Net tackles 3D visual grounding by explicitly modeling relative spatial relations in 3D space through a novel 3D Relative Position Multi-head Attention (3DRP-MA) within a transformer-based one-stage framework. It introduces a piecewise distance encoding and multi-directional relative distances (D_x, D_y, D_z, D_e) to capture nuanced object-pair relations, and a soft-labeling strategy to stabilize supervision across multiple key points. The approach delivers state-of-the-art results on ScanRefer, Nr3D, and Sr3D, with ablations confirming the contributions of 3DRP-MA, relative encodings, and soft-labeling to improved accuracy and robustness. This work advances practical 3D grounding by enabling precise relation reasoning directly in a single-stage pipeline, which can benefit robotics, augmented reality, and interactively described scene understanding.

Abstract

3D visual grounding aims to localize the target object in a 3D point cloud by a free-form language description. Typically, the sentences describing the target object tend to provide information about its relative relation between other objects and its position within the whole scene. In this work, we propose a relation-aware one-stage framework, named 3D Relative Position-aware Network (3DRP-Net), which can effectively capture the relative spatial relationships between objects and enhance object attributes. Specifically, 1) we propose a 3D Relative Position Multi-head Attention (3DRP-MA) module to analyze relative relations from different directions in the context of object pairs, which helps the model to focus on the specific object relations mentioned in the sentence. 2) We designed a soft-labeling strategy to alleviate the spatial ambiguity caused by redundant points, which further stabilizes and enhances the learning process through a constant and discriminative distribution. Extensive experiments conducted on three benchmarks (i.e., ScanRefer and Nr3D/Sr3D) demonstrate that our method outperforms all the state-of-the-art methods in general. The source code will be released on GitHub.

Figures (7)

• Figure 1: 3D visual grounding is the task of grounding a description in a 3D scene. In the sentences, all the words indicating the relative positions of the target object are bolded. Notice that relative position relations between objects are crucial for distinguishing the target object, and the relative position-related descriptions in 3D space are complex (e.g., "above", "on the left", "in front of", and "next to", etc.)
• Figure 2: 3DRP-Net is a transformer-based one-stage 3D VG model which takes a 3D point cloud and a description as inputs and outputs the bounding box of the object most relevant to the input expression. In the stacked transformer layer, the 3DRP-MA captures the relative relations between points in the 3D perspective. Specifically, the two self-attentions based on 3DRP-MA capture the relative relations between objects, while the cross-attention between key points and seed points enhances the global position information.
• Figure 3: Comparison of various labeling strategies.
• Figure 4: The visualization results of some success cases. The blue/green/red colors indicate the ground truth/correct/incorrect boxes.
• Figure 5: The visualization results of some failure cases. The ground-truth boxes are labeled in blue and the incorrectly predicted boxes are marked in red.