Breaking Self-Attention Failure: Rethinking Query Initialization for Infrared Small Target Detection

TL;DR

Infrared small target detection is challenged by low SNR and cluttered backgrounds, causing DETR-based transformers to dilute target embeddings and struggle with precise localization. The authors introduce SEF-DETR, a threefold framework that leverages frequency-domain priors via Frequency-guided Patch Screening, enhances multi-scale embeddings with Dynamic Embedding Enhancement, and selects robust queries through Reliability-Consistency-aware Fusion. Across three public IRSTD datasets, SEF-DETR delivers state-of-the-art performance, especially for very tiny targets, with minimal computational overhead. This approach offers a practical, efficient enhancement to DETR-based IRSTD by mitigating embedding dilution and improving query initialization in challenging infrared scenes.

Abstract

Infrared small target detection (IRSTD) faces significant challenges due to the low signal-to-noise ratio (SNR), small target size, and complex cluttered backgrounds. Although recent DETR-based detectors benefit from global context modeling, they exhibit notable performance degradation on IRSTD. We revisit this phenomenon and reveal that the target-relevant embeddings of IRST are inevitably overwhelmed by dominant background features due to the self-attention mechanism, leading to unreliable query initialization and inaccurate target localization. To address this issue, we propose SEF-DETR, a novel framework that refines query initialization for IRSTD. Specifically, SEF-DETR consists of three components: Frequency-guided Patch Screening (FPS), Dynamic Embedding Enhancement (DEE), and Reliability-Consistency-aware Fusion (RCF). The FPS module leverages the Fourier spectrum of local patches to construct a target-relevant density map, suppressing background-dominated features. DEE strengthens multi-scale representations in a target-aware manner, while RCF further refines object queries by enforcing spatial-frequency consistency and reliability. Extensive experiments on three public IRSTD datasets demonstrate that SEF-DETR achieves superior detection performance compared to state-of-the-art methods, delivering a robust and efficient solution for infrared small target detection task.

Figures (6)

• Figure 1: Comparison of three different query initialization methods. (a) The static queries are used for each inference. (b) Select queries from the output of encoder. (c) Our proposed query initialization with screening, enhancement, and fusion mechanism.
• Figure 2: Illustration of a local patch of IRST from IRSTD-1k dataset. We compute the complete FFT spectrum of each local patch. The red, yellow, and gray boxes denote the target, target-like interference, and background regions, respectively.
• Figure 3: Comparison of similarity $s$ in the "$l=0$ to $l=6$" layers on IRSTD-1k, NUAA-SIRST and NUDT-SIRST dataset. It can be seen that the target-relevant embedding in the deeper layers are gradually diluted by the background-relevant embedding.
• Figure 4: Overview of our proposed SEF-DETR. The input infrared image is processed through two complementary paths. The top branch shows the Frequency-guided Patch Screening (FPS) Module, which produces a pixel-wise target-relevant density map indicating potential target regions. This map is then employed at two critical stages: in the Dynamic Embedding Enhancement (DEE) Module to refine multi-scale embedding features, and in the Reliability-Consistency-aware Fusion (RCF) Module to guide the selection of object queries. Finally, these refined queries are fed into the Transformer Decoder to perform accurate target localization and classification.
• Figure 5: Visualization comparison of detection results via different methods on representative images from IRSTD-1k datasets, indicate the land, forests and skies interfere. The red, yellow, and blue boxes denote correct detection, false alarms, and missed detections, respectively.