E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection

TL;DR

E2E-MFD proposes a unified end-to-end framework for synchronous multimodal fusion and object detection, addressing inefficiencies of cascaded MF-OD pipelines. It couples ORPPT for coarse-to-fine, multi-granularity feature extraction with CFDP as a diffusion-based detection head, and stabilizes training via Gradient Matrix Task-Alignment (GMTA). The approach achieves strong fusion quality and improved detection performance on multiple visible–infrared datasets, including reported mAP50 gains of 3.9% on M3FD and 2.0% on DroneVehicle over state-of-the-art. These results demonstrate that end-to-end joint optimization with gradient alignment can yield both visually appealing fusion and robust downstream object detection, with practical implications for autonomous systems.

Abstract

Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Addressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint optimization across components to avoid suboptimal solutions tied to individual tasks. Furthermore, it implements a comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such as a 3.9% and 2.0% mAP50 increase on horizontal object detection dataset M3FD and oriented object detection dataset DroneVehicle, respectively, compared to state-of-the-art approaches. The code is released at https://github.com/icey-zhang/E2E-MFD.

Figures (12)

• Figure 1: Comparison of (d) E2E-MFD with existing MF-OD task paradigms (a) Two-Stage (Separate Cascaded), (b) Two-stage (Joint Cascaded) and (c) Multi-stage (Joint Cascaded).
• Figure 2: An overview of the proposed E2E-MFD framework, which consists of a backbone, nodes, and branches. The backbone is utilized to extract multimodal image features. A fine-grained fusion network (ORPPT) and diffusion-based object detection network (CFDP) are optimized by synchronous joint optimization (GMTA) in an end-to-end manner.
• Figure 3: Visual results of image fusion on M3FD.
• Figure 4: Visual results of object detection on M3FD.
• Figure 5: Visualization of task dominance and conflicting gradients in joint learning of OD and MF.