Fine-Grained Spatial and Verbal Losses for 3D Visual Grounding

TL;DR

Two novel losses for 3D visual grounding are introduced: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description.

Abstract

3D visual grounding consists of identifying the instance in a 3D scene which is referred by an accompanying language description. While several architectures have been proposed within the commonly employed grounding-by-selection framework, the utilized losses are comparatively under-explored. In particular, most methods rely on a basic supervised cross-entropy loss on the predicted distribution over candidate instances, which fails to model both spatial relations between instances and the internal fine-grained word-level structure of the verbal referral. Sparse attempts to additionally supervise verbal embeddings globally by learning the class of the referred instance from the description or employing verbo-visual contrast to better separate instance embeddings do not fundamentally lift the aforementioned limitations. Responding to these shortcomings, we introduce two novel losses for 3D visual grounding: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description. In addition, we equip the verbo-visual fusion module of our new 3D visual grounding architecture AsphaltNet with a top-down bidirectional attentive fusion block, which enables the supervisory signals from our two losses to propagate to the respective converse branches of the network and thus aid the latter to learn context-aware instance embeddings and grounding-aware verbal embeddings. AsphaltNet proposes novel auxiliary losses to aid 3D visual grounding with competitive results compared to the state-of-the-art on the ReferIt3D benchmark.

Figures (8)

• Figure 1: We propose two auxiliary losses to aid the task of 3D visual grounding: i) offset loss $\mathcal{L}_o$ between instances to promote better localization and improve object separability; ii) span loss $\mathcal{L}_{sp}$ on the prompts to provide a higher granularity of supervision.
• Figure 2: Illustration of the AsphaltNet pipeline. Via a UNet, we first compute per point features from the 3D point cloud and construct instance tokens via mean pooling on the provided instance masks and concatenation with the corresponding bounding box centroid and mean color vector. We encode the verbal input and generate word tokens using a pretrained LLM. A top-down bidirectional attentive fusion module consumes the instance and word tokens to ground the natural language description within the scene, i.e. predict the referred object. To aid the training process, we employ two auxiliary losses: $\mathcal{L}_o$ that aids localization, and $\mathcal{L}_{sp}$ to supervise the language encoding.
• Figure 3: Illustration of top-down bidirectional attentive fusion.
• Figure 4: Sample offset predictions of AsphaltNet from the Nr3D achlioptas2020referit_3dval-set given the prompt "Facing the armchairs, the chair in the upper left". We overlay the heatmap of the predicted offset values per instance to the target on top of the scene for all the three layers of the fusion module. The offset prediction is refined at each step as the distribution of predicted offsets tightens and converges on the target instance.
• Figure 5: Qualitative results on the Nr3D val-set comparing AsphaltNet with the baseline ConcreteNet model. We illustrate the predicted instances via instance masks (red) for ConcreteNet, and (blue) for AsphaltNet. In green, we provide the axis aligned bounding box of the referred ground truth object.