Disentangle Object and Non-object Infrared Features via Language Guidance

TL;DR

This work tackles infrared object detection under challenging lighting by introducing LGFD, a vision-language representation learning framework that disentangles object and non-object infrared features using textual guidance. Central to LGFD are the Semantic Feature Alignment (SFA) and Object Feature Disentanglement (OFD) modules, which align object features with auxiliary captions and minimize correlation between object and non-object streams, respectively, within a joint loss $L = L_{det} + \alpha L_{al} + \beta L_{ds}$. Auxiliary captions are generated via a rule-based Bbox2Caption method, enabling text supervision without extra labeling. LGFD achieves state-of-the-art performance on FLIR and M3FD without using paired RGB data, demonstrating the practical impact of language-guided disentanglement for infrared perception and suggesting broad applicability to other multimodal tasks.

Abstract

Infrared object detection focuses on identifying and locating objects in complex environments (\eg, dark, snow, and rain) where visible imaging cameras are disabled by poor illumination. However, due to low contrast and weak edge information in infrared images, it is challenging to extract discriminative object features for robust detection. To deal with this issue, we propose a novel vision-language representation learning paradigm for infrared object detection. An additional textual supervision with rich semantic information is explored to guide the disentanglement of object and non-object features. Specifically, we propose a Semantic Feature Alignment (SFA) module to align the object features with the corresponding text features. Furthermore, we develop an Object Feature Disentanglement (OFD) module that disentangles text-aligned object features and non-object features by minimizing their correlation. Finally, the disentangled object features are entered into the detection head. In this manner, the detection performance can be remarkably enhanced via more discriminative and less noisy features. Extensive experimental results demonstrate that our approach achieves superior performance on two benchmarks: M\textsuperscript{3}FD (83.7\% mAP), FLIR (86.1\% mAP). Our code will be publicly available once the paper is accepted.

Figures (7)

• Figure 1: Comparison between previous method and our LGFD for feature disentanglement. (a) The previous method typically struggle to extract object features through convolution in an entangle feature space. (b) Our LGFD adopts a language-guided representation learning strategy. It disentangles the object and non-object feature via textual semantic information guidance.
• Figure 1: Statistical analysis of objects quantity and sensitivity analysis of channel decomposition. (a) The quantity of objects is calculated from different FPN layers on the two datasets. (b) The horizontal axis represents the proportion of decomposed object feature compared to the original feature.
• Figure 2: Our LGFD’s overall architecture. Initially, BBox2Caption is introduced to generate the detailed descriptions in a rule-based manner. The paired image-text data are entered into encoders to obtain features. Image FPN features are separated into object and non-object parts. Subsequently, the SFA module aligns object and text features to make the model learn semantic information pertinent to the objects. Meanwhile, the OFD module minimizes the similarity between the object and non-object features via a constraint loss. It can mitigate the influence of background on object feature extraction. Ultimately, the robust object features are leveraged to produce final detection output.
• Figure 2: An example of natural-language description.
• Figure 3: Visualization of feature activations results of our LGFD and baseline on FLIR with ground truth (GT). Orange (a-c) and red (d-f) boxes indicate weak and false activations, respectively. Green boxes represent the results of LGFD for comparison.