GraphFusion3D: Dynamic Graph Attention Convolution with Adaptive Cross-Modal Transformer for 3D Object Detection

TL;DR

GraphFusion3D presents a multimodal 3D object detector that fuses point clouds and RGB images through a Graph Reasoning Module for relational context, an Adaptive Cross-Modal Transformer for dynamic image-point alignment, and a Progressive Cascaded Refinement Decoder for multi-stage localization. The method demonstrates strong performance on SUN RGB-D and competitive results on ScanNetV2, highlighting the value of context-aware fusion and hierarchical reasoning in indoor scenes. Key contributions include multi-scale graph attention for proposal context, modality gating for robust cross-modal fusion, and a cascaded decoder that progressively improves detection quality. Together, these components advance robust 3D perception in cluttered, indoor environments with partial observations.

Abstract

Despite significant progress in 3D object detection, point clouds remain challenging due to sparse data, incomplete structures, and limited semantic information. Capturing contextual relationships between distant objects presents additional difficulties. To address these challenges, we propose GraphFusion3D, a unified framework combining multi-modal fusion with advanced feature learning. Our approach introduces the Adaptive Cross-Modal Transformer (ACMT), which adaptively integrates image features into point representations to enrich both geometric and semantic information. For proposal refinement, we introduce the Graph Reasoning Module (GRM), a novel mechanism that models neighborhood relationships to simultaneously capture local geometric structures and global semantic context. The module employs multi-scale graph attention to dynamically weight both spatial proximity and feature similarity between proposals. We further employ a cascade decoder that progressively refines detections through multi-stage predictions. Extensive experiments on SUN RGB-D (70.6\% AP$_{25}$ and 51.2\% AP$_{50}$) and ScanNetV2 (75.1\% AP$_{25}$ and 60.8\% AP$_{50}$) demonstrate a substantial performance improvement over existing approaches.

Figures (5)

• Figure 1: Our proposed GraphFusion3D architecture. The framework processes point clouds and RGB images through four key stages: (1) feature extraction via point and image backbone networks, (2) contextual refinement using the Graph Reasoning module, (3) multi-modal fusion via Adaptive Cross-Modal Transformer, and (4) progressive Cascade Refinement Decoding. Final detections are generated after multi-stage refinement.
• Figure 2: The architecture of the Adaptive Cross-Modal Transformer module.
• Figure 3: Illustration of a progressive cascaded refinement decoder module. Left: Bounding box from a single proposal point with predicted $\delta$. Right: Updating process from proposal point to bounding box center.
• Figure 4: Qualitative prediction results on the ScanNetV2. The first row displays the ground-truth data, while the second row presents the detection results from our GraphFusion3D.
• Figure 5: Qualitative prediction results on the SUN RGB-D. The first row displays the ground-truth data, while the second row presents the detection results from our GraphFusion3D.