Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation

TL;DR

This work proposes HCF-RES, a multi-modal framework with two key innovations: Hierarchical Visual Semantic Decomposition leverages SAM instance masks to guide CLIP encoding at dual granularities -- pixel-level and instance-level features -- preserving object boundaries during 2D-to-3D projection.

Abstract

Generalised 3D Referring Expression Segmentation (3D-GRES) localizes objects in 3D scenes based on natural language, even when descriptions match multiple or zero targets. Existing methods rely solely on sparse point clouds, lacking rich visual semantics for fine-grained descriptions. We propose HCF-RES, a multi-modal framework with two key innovations. First, Hierarchical Visual Semantic Decomposition leverages SAM instance masks to guide CLIP encoding at dual granularities -- pixel-level and instance-level features -- preserving object boundaries during 2D-to-3D projection. Second, Progressive Multi-level Fusion integrates representations through intra-modal collaboration, cross-modal adaptive weighting between 2D semantic and 3D geometric features, and language-guided refinement. HCF-RES achieves state-of-the-art results on both ScanRefer and Multi3DRefer.

Figures (4)

• Figure 1: (a) Language descriptions inherently encode hierarchical object semantics (e.g., main object "chair" with attributes "gray" and relationship "below desk"). (b) Traditional methods rely solely on sparse point clouds, leading to wrong segmentation. (c) Our method leverages pre-trained vision models (CLIPclip and SAMSamv1) on filtered multi-view images to extract rich semantic features, achieving better alignment across images, point clouds, and natural language.
• Figure 2: Overview of our Hierarchical Visual Semantic Decomposition: We employ the SAM Samv1 to segment the instances segmentation masks for each multi-view images without requiring annotations and each mask is then filtered by quality before encoding with CLIP clip to obtain its instance-level and pixel-level features. These multi-granularity features are then subsequently projected to the point cloud and aggregated into superpoints representations.
• Figure 3: Pipeline of HCF-RES: The framework processes three input modalities: point clouds via 3D U-Net, text via RoBERTa, and multi-view images via CLIP guided by SAM * through our Hierarchical Visual Semantic Decomposition (Section \ref{['SAM-based extraction']} and Figure \ref{['fig:multi-view']}). After cross-modal feature integration and language-guided sampling, our Language-guided Instance Refinement enhances selected queries through scene context awareness and 2D semantic fusion. Enhanced queries are decoded via a 6-layer decoder base_refer:2 for final 3D referring expression segmentation.
• Figure 4: Qualitative Result to compare our method HCF-RES with the state-of-art MDIN base_refer:2 and IPDNbase_refer:3. Overall, our method produces more accurate segmentations. While IPDN achieves comparable results in general, it often shows noisy predictions near object boundaries.