Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection

TL;DR

Open-vocabulary 3D object detection requires recognizing unseen categories while leveraging scene context. HCMA presents a hierarchical cross-modal framework that fuses object-level and global scene context through Hierarchical Data Integration, Interactive Cross-Modal Alignment, and Object-Focusing Context Adjustment. A hierarchical contrastive objective aligns 3D, image, and text features across object, view, and scene levels, aided by top-down scene representations and a Noise-Injection Enhancement block. Results on ScanNet and SUN RGB-D achieve state-of-the-art OV-3DOD performance, with strong cross-dataset transfer and open-vocabulary generalization, underscoring the value of scene context in open-world 3D understanding and offering efficient multi-level cross-modal fusion for 3D perception.

Abstract

Open-vocabulary 3D object detection (OV-3DOD) aims at localizing and classifying novel objects beyond closed sets. The recent success of vision-language models (VLMs) has demonstrated their remarkable capabilities to understand open vocabularies. Existing works that leverage VLMs for 3D object detection (3DOD) generally resort to representations that lose the rich scene context required for 3D perception. To address this problem, we propose in this paper a hierarchical framework, named HCMA, to simultaneously learn local object and global scene information for OV-3DOD. Specifically, we first design a Hierarchical Data Integration (HDI) approach to obtain coarse-to-fine 3D-image-text data, which is fed into a VLM to extract object-centric knowledge. To facilitate the association of feature hierarchies, we then propose an Interactive Cross-Modal Alignment (ICMA) strategy to establish effective intra-level and inter-level feature connections. To better align features across different levels, we further propose an Object-Focusing Context Adjustment (OFCA) module to refine multi-level features by emphasizing object-related features. Extensive experiments demonstrate that the proposed method outperforms SOTA methods on the existing OV-3DOD benchmarks. It also achieves promising OV-3DOD results even without any 3D annotations.

Figures (6)

• Figure 1: Our HCMA framework is a three-stream network containing pre-trained CLIP image and text encoders, a 3D detector, and our proposed Object-Focusing Context Adjustment (OFCA) module. Each stream processes three types of hierarchical semantics. The input data of each stream is derived from our Hierarchical Data Integration (HDI) approach, while the output semantics of each stream is associated by ourInteractive Cross-Modal Alignment (ICMA) strategy.
• Figure 2: Illustration of the Hierarchical Data Integration (HDI) approach. HDI introduces object-level, view-level, and scene-level hierarchies to associate point clouds, images, and texts.
• Figure 3: Construction of the positive and negative samples based on the scene label.
• Figure 4: Structure of the proposed Object-Focusing Context Adjustment (OFCA) module.
• Figure 5: Qualitative comparison with OV-3DET. Our HCMA framework can perform more accurate OV-3DOD and can detect a wider vocabulary of objects in 3D scenes. For each case, the detection text prompts are shown on the left.