AG-Fusion: adaptive gated multimodal fusion for 3d object detection in complex scenes

TL;DR

AG-Fusion introduces a robust multimodal fusion approach for 3D object detection by combining window-based self-attention to enhance per-modality BEV features with a bidirectional cross-attention mechanism and a content-adaptive gating fusion. The method includes a multi-level feature aggregation step to produce a unified BEV representation, enabling reliable detection in challenging industrial scenes. A new Excavator3D dataset is proposed to benchmark performance under occlusion, noise, and clutter, complementing KITTI evaluations. Empirical results on KITTI and E3D demonstrate state-of-the-art accuracy and notable robustness gains, highlighting the practical impact for industrial perception and heavy-equipment automation.

Abstract

Multimodal camera-LiDAR fusion technology has found extensive application in 3D object detection, demonstrating encouraging performance. However, existing methods exhibit significant performance degradation in challenging scenarios characterized by sensor degradation or environmental disturbances. We propose a novel Adaptive Gated Fusion (AG-Fusion) approach that selectively integrates cross-modal knowledge by identifying reliable patterns for robust detection in complex scenes. Specifically, we first project features from each modality into a unified BEV space and enhance them using a window-based attention mechanism. Subsequently, an adaptive gated fusion module based on cross-modal attention is designed to integrate these features into reliable BEV representations robust to challenging environments. Furthermore, we construct a new dataset named Excavator3D (E3D) focusing on challenging excavator operation scenarios to benchmark performance in complex conditions. Our method not only achieves competitive performance on the standard KITTI dataset with 93.92% accuracy, but also significantly outperforms the baseline by 24.88% on the challenging E3D dataset, demonstrating superior robustness to unreliable modal information in complex industrial scenes.

Figures (4)

• Figure 1: Overview Of The Proposed AG-Fusion Architecture.
• Figure 2: Structure of adaptive gated block.
• Figure 3: Example scenes and distribution of minor/severe occlusion and truncation in the E3D dataset.
• Figure 4: Performance comparison between the proposed fusion method and BEVFusion on the E3D dataset.