DPDETR: Decoupled Position Detection Transformer for Infrared-Visible Object Detection

TL;DR

DPDETR addresses modality misalignment in infrared-visible object detection by explicitly decoupling object positions in infrared and visible modalities and aligning cross-modal features with a Decoupled Position Multispectral Deformable cross-attention. The approach introduces a Paired IoU-aware Competitive Query Selection, a Decoupled Multispectral Decoder with decoupled cross-attention branches, and a Decoupled Position Contrastive DeNoising Training regime to robustly learn dual-modality correspondences. Empirically, DPDETR achieves state-of-the-art results on DroneVehicle and KAIST, with strong ablations confirming the benefits of decoupled positions, denoising training, and query decoupling, while maintaining competitive computational costs. The work advances reliable multispectral detection under challenging misalignment scenarios, enabling robust performance in real-world infrared-visible sensing tasks.

Abstract

Infrared-visible object detection aims to achieve robust object detection by leveraging the complementary information of infrared and visible image pairs. However, the commonly existing modality misalignment problem presents two challenges: fusing misalignment complementary features is difficult, and current methods cannot reliably locate objects in both modalities under misalignment conditions. In this paper, we propose a Decoupled Position Detection Transformer (DPDETR) to address these issues. Specifically, we explicitly define the object category, visible modality position, and infrared modality position to enable the network to learn the intrinsic relationships and output reliably positions of objects in both modalities. To fuse misaligned object features reliably, we propose a Decoupled Position Multispectral Cross-attention module that adaptively samples and aggregates multispectral complementary features with the constraint of infrared and visible reference positions. Additionally, we design a query-decoupled Multispectral Decoder structure to address the the conflict in feature focus among the three kinds of object information in our task and propose a Decoupled Position Contrastive DeNoising Training strategy to enhance the DPDETR's ability to learn decoupled positions. Experiments on DroneVehicle and KAIST datasets demonstrate significant improvements compared to other state-of-the-art methods. The code will be released at https://github.com/gjj45/DPDETR

Figures (10)

• Figure 1: Misalignment problem in infrared-visible object detection and the detection comparison between the common method and our method. (a) Examples of modality misalignment in infrared and visible image pairs. The yellow and red boxes represent the position of the same object in the infrared and visible modalities, respectively. The yellow dashed line represents the object's position in the infrared modality. (b) An example of w/o DP (without Decoupled Position) method where the detection results make it difficult to distinguish the same object in two modalities due to misalignment. Our method can reliably locate objects in both modalities and identify the same object with the same ID.
• Figure 2: The overall architecture of DPDETR. The framework consists of feature extraction and encoding modules, a Paired IoU-aware Competitive Query Selection module, a Decoupled Multispectral Decoder, and the Decoupled Position Contrastive DeNoising Training strategy. The final output of DPDETR is the category of each object and its reliable position in both visible and infrared images.
• Figure 3: The structure of the Decoupled Multispectral Transformer Decoder (DeNoising Training Group is omitted in the figure) and Decoupled Position Multispectral Deformable cross-attention module.
• Figure 4: Demonstration of adding noise in terms of shift, scale, and angle to a pair of ground truth boxes. Red, light yellow, and dark yellow boxes represent the infrared GT box, positive infrared denoising box, and negative infrared denoising box, respectively. Blue, light orange, and dark orange boxes represent the visible GT box, positive visible denoising box, and negative visible denoising box, respectively.
• Figure 5: The structure of DPCDN group and the useage of positive and negative denoising queries.