FD2-Net: Frequency-Driven Feature Decomposition Network for Infrared-Visible Object Detection

TL;DR

FD2-Net addresses IVOD by explicitly exploiting frequency characteristics across infrared and visible modalities. It introduces a Feature Decomposition Encoder with a $2D$ DCT-based High-Frequency Unit and a multi-scale Low-Frequency Context Refinement, coupled with a parameter-free Complementary Strengths Strategy to recouple modalities. A Multimodal Reconstruction Mechanism, featuring feature-level masks and a Cross-Reconstruction Unit, recovers lost details and enhances discriminative representation. On LLVIP, FLIR, and M3FD benchmarks, FD2-Net achieves state-of-the-art $mAP_{50}$ (e.g., $96.2\%$ on LLVIP, $82.9\%$ on FLIR, and $83.5\%$ on M3FD), validating the effectiveness of frequency-driven decomposition and reconstruction for robust IVOD performance.

Abstract

Infrared-visible object detection (IVOD) seeks to harness the complementary information in infrared and visible images, thereby enhancing the performance of detectors in complex environments. However, existing methods often neglect the frequency characteristics of complementary information, such as the abundant high-frequency details in visible images and the valuable low-frequency thermal information in infrared images, thus constraining detection performance. To solve this problem, we introduce a novel Frequency-Driven Feature Decomposition Network for IVOD, called FD2-Net, which effectively captures the unique frequency representations of complementary information across multimodal visual spaces. Specifically, we propose a feature decomposition encoder, wherein the high-frequency unit (HFU) utilizes discrete cosine transform to capture representative high-frequency features, while the low-frequency unit (LFU) employs dynamic receptive fields to model the multi-scale context of diverse objects. Next, we adopt a parameter-free complementary strengths strategy to enhance multimodal features through seamless inter-frequency recoupling. Furthermore, we innovatively design a multimodal reconstruction mechanism that recovers image details lost during feature extraction, further leveraging the complementary information from infrared and visible images to enhance overall representational capacity. Extensive experiments demonstrate that FD2-Net outperforms state-of-the-art (SOTA) models across various IVOD benchmarks, i.e. LLVIP (96.2% mAP), FLIR (82.9% mAP), and M3FD (83.5% mAP).

Figures (4)

• Figure 1: Illustration of the differences between our FD2-Net and existing IVOD approaches. Our algorithm employs frequency decoupling to separate high- and low-frequency information in infrared and visible images, thereby effectively leveraging multimodal complementary features to extract more discriminative and robust characteristics.
• Figure 2: The architecture (top row) and core components (bottom row) of our FD2-Net. It has three components: (1) Feature Decomposition Encoder, which effectively extracts high/low-frequency features in multimodal visual space. (2) Multimodal Reconstruction Mechanism, which further learns the distinguishing and complementary features of each modality through the reconstruction of multimodal images to enhance feature representation. (3) Multi-Scale Detection Head, which uses visual features from (1) and (2) to complete object classification and localization.
• Figure 3: Visual comparison of FD2Net with 10 SOTA methods. Green boxes are detection results, while red dashed boxes mark missed objects (false negatives).
• Figure 4: Left: Features from the original YOLOv5n, Right: Features from the proposed FD2Net.