SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection

TL;DR

An accurate and efficient multimodal object detection method named SeaDATE is introduced and a contrastive learning module aimed at learning features of multimodal samples is designed, remedying the shortcomings of Transformer-guided fusion in extracting deep semantic features, and effectively utilizing cross-modal information.

Abstract

Multimodal object detection leverages diverse modal information to enhance the accuracy and robustness of detectors. By learning long-term dependencies, Transformer can effectively integrate multimodal features in the feature extraction stage, which greatly improves the performance of multimodal object detection. However, current methods merely stack Transformer-guided fusion techniques without exploring their capability to extract features at various depth layers of network, thus limiting the improvements in detection performance. In this paper, we introduce an accurate and efficient object detection method named SeaDATE. Initially, we propose a novel dual attention Feature Fusion (DTF) module that, under Transformer's guidance, integrates local and global information through a dual attention mechanism, strengthening the fusion of modal features from orthogonal perspectives using spatial and channel tokens. Meanwhile, our theoretical analysis and empirical validation demonstrate that the Transformer-guided fusion method, treating images as sequences of pixels for fusion, performs better on shallow features' detail information compared to deep semantic information. To address this, we designed a contrastive learning (CL) module aimed at learning features of multimodal samples, remedying the shortcomings of Transformer-guided fusion in extracting deep semantic features, and effectively utilizing cross-modal information. Extensive experiments and ablation studies on the FLIR, LLVIP, and M3FD datasets have proven our method to be effective, achieving state-of-the-art detection performance.

Figures (7)

• Figure 1: Differernt strategies for fusion.
• Figure 2: Overview framework of the proposed SeaDATE. The network architecture is bifurcated into three components: 1) The feature extraction module with dual-stream E-ELAN as the backbone network. 2) DTF module for multi-modal feature fusion. 3) CL module for remedying Transformer fusion method.
• Figure 3: Model architecture for our dual attention block. In this work, we use these two types of attention to obtain both local fine-grained and global features.
• Figure 4: Proposed CL module.
• Figure 5: Qualitative comparison of multimodal object detection in the FLIR dataset. First column: ground truth, second column: detection results of the baseline, third column: detection results of our method. Note that yellow inverted triangles indicate FPs, and red inverted triangles show FNs. Zoomed in to see details.