RefMask3D: Language-Guided Transformer for 3D Referring Segmentation

TL;DR

This paper tackles 3D referring segmentation by introducing RefMask3D, an end-to-end single-stage framework that fuses vision and language throughout encoding and decoding. It contributes three novel components: Geometry-Enhanced Group-Word Attention for geometry-aware cross-modal fusion, Linguistic Primitives Construction for semantic attribute representation, and Object Cluster Module for holistic object reasoning. Together, these modules yield state-of-the-art results on 3D referring segmentation, 3D visual grounding, and 2D referring image segmentation, notably surpassing prior methods on the ScanRefer benchmark. The approach advances multimodal understanding in sparse 3D data and demonstrates strong generalization to related tasks, highlighting its practical impact for accurate language-guided scene understanding.

Abstract

3D referring segmentation is an emerging and challenging vision-language task that aims to segment the object described by a natural language expression in a point cloud scene. The key challenge behind this task is vision-language feature fusion and alignment. In this work, we propose RefMask3D to explore the comprehensive multi-modal feature interaction and understanding. First, we propose a Geometry-Enhanced Group-Word Attention to integrate language with geometrically coherent sub-clouds through cross-modal group-word attention, which effectively addresses the challenges posed by the sparse and irregular nature of point clouds. Then, we introduce a Linguistic Primitives Construction to produce semantic primitives representing distinct semantic attributes, which greatly enhance the vision-language understanding at the decoding stage. Furthermore, we introduce an Object Cluster Module that analyzes the interrelationships among linguistic primitives to consolidate their insights and pinpoint common characteristics, helping to capture holistic information and enhance the precision of target identification. The proposed RefMask3D achieves new state-of-the-art performance on 3D referring segmentation, 3D visual grounding, and also 2D referring image segmentation. Especially, RefMask3D outperforms previous state-of-the-art method by a large margin of 3.16% mIoU} on the challenging ScanRefer dataset. Code is available at https://github.com/heshuting555/RefMask3D.

Figures (5)

• Figure 1: (a) Two-stage framework, fusing language features in the later matching stage, exhibit limited interactions and weak alignment between vision and language features. In contrast, (b) our RefMask3D conducts comprehensive vision-language fusion in both the early feature encoding stage and decoding stage. Combined with contrastive learning, our model learns a well-structured vision-language joint feature space than two-stage methods.
• Figure 2: The framework overview of the proposed RefMask3D. It extracts text-enriched point features from the point encoder which is assisted by Geometry-Enhanced Group-Word Attention. Subsequently, the Linguistic Primitives Construction Module generates primitives to embody specific semantic attributes. These primitives are then fed into the Transformer Decoder to focus on diverse semantics. Object Cluster Module is employed to analyze the interrelationships among linguistic primitives to unify their insights and pinpoint common characteristics to enhance the precision of target identification.
• Figure 3: Linguistic Primitives Construction (LPC) initializes various primitives $\mathcal{O}$ to express distinct semantic attributes. These primitives after interacting with linguistic information are capable of acquiring corresponding attribute values denoted $\mathcal{O}'$.
• Figure 4: Primitives heatmap visualization. Different primitives represent distinct semantic attributes. Blue indicates the lowest response levels, while red signifies the highest response levels.
• Figure 5: Visualization results of complex language descriptions on ScanRefer. we use color masks for clarity: green represents the ground truth, red indicates incorrect predictions by TGNN, and blue signifies correct predictions by ours.