Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding

TL;DR

Dense 3D visual grounding is addressed by predicting fine-grained 3D masks rather than boxes in referred-object scenarios. ConcreteNet combines a kernel-based 3D instance segmentation backbone with a transformer-based language encoder and a quartet of fusion enhancements: bottom-up attentive fusion, verbo-visual contrastive learning, a global camera token to handle view-dependence, and multi-view ensembling. The approach achieves state-of-the-art results on the ScanRefer benchmark and won the ICCV workshop challenge, demonstrating improved disambiguation for repetitive instances and higher-quality masks. This work advances practical 3D grounding for tasks requiring precise geometry, such as manipulation and interaction in real-world environments.

Abstract

3D visual grounding is the task of localizing the object in a 3D scene which is referred by a description in natural language. With a wide range of applications ranging from autonomous indoor robotics to AR/VR, the task has recently risen in popularity. A common formulation to tackle 3D visual grounding is grounding-by-detection, where localization is done via bounding boxes. However, for real-life applications that require physical interactions, a bounding box insufficiently describes the geometry of an object. We therefore tackle the problem of dense 3D visual grounding, i.e. referral-based 3D instance segmentation. We propose a dense 3D grounding network ConcreteNet, featuring four novel stand-alone modules that aim to improve grounding performance for challenging repetitive instances, i.e. instances with distractors of the same semantic class. First, we introduce a bottom-up attentive fusion module that aims to disambiguate inter-instance relational cues, next, we construct a contrastive training scheme to induce separation in the latent space, we then resolve view-dependent utterances via a learned global camera token, and finally we employ multi-view ensembling to improve referred mask quality. ConcreteNet ranks 1st on the challenging ScanRefer online benchmark and has won the ICCV 3rd Workshop on Language for 3D Scenes "3D Object Localization" challenge. Our code is available at ouenal.github.io/concretenet/.

Figures (6)

• Figure 1: ConcreteNet localizes referred objects via dense masks rather than boxes.
• Figure 2: Illustration of our ConcreteNet dense 3D visual grounding pipeline (left). Given a point cloud and a natural language prompt, we first generate instance candidates (blue) and word embeddings (pink). We then fuse these to densely ground the verbal description to the 3D scene. We improve performance by localizing attention via a bottom-up attentive fusion module (right), utilizing contrastive learning to promote better feature separability, and learning the camera position to disambiguate view-dependent descriptions. Our final prediction is generated by merging the token of the best-fitting instance with its predicted mask.
• Figure 3: Qualitative results from the ScanRefer val-set depicting the dense predictions of ConcreteNet against the ground truth and 3DVG-Transformer zhao20213dvgt predictions. We showcase two cases that illustrate the effectiveness of BAF (left) and GCT (right).
• Figure 4: Additional qualitative result from the ScanRefer val-set showing the benefits of a learned global camera token.
• Figure 5: Failure case from the ScanRefer val-set. While the predictions from both ConcreteNet and 3DVG-Transformer do not match the ground truth, given the symmetric nature of the scene along with the vagueness of the description, it can be seen that the cue does match both predictions and the ground truth.