Spatio-Temporal Context Learning with Temporal Difference Convolution for Moving Infrared Small Target Detection

TL;DR

This work tackles moving infrared small target detection (IRSTD), where targets are tiny and embedded in cluttered, dynamic infrared scenes. It introduces TDCNet, a multi-branch architecture that combines a temporal difference convolution (TDC) backbone with a TDCR module for multi-scale motion-context modeling and a TDC-guided spatio-temporal attention (TDCSTA) to fuse motion-aware features with parallel 3D features. The TDCR module enables explicit short-, mid-, and long-term motion cues, with a re-parameterization that maintains efficiency during inference, while TDCSTA performs cross-stream attention to refine crucial target regions. Across IRSTD-UAV and IRDST benchmarks, TDCNet achieves state-of-the-art detection performance, robustly handling low SCR and complex backgrounds, underscoring its practical potential for UAV surveillance and infrared target detection tasks.

Abstract

Moving infrared small target detection (IRSTD) plays a critical role in practical applications, such as surveillance of unmanned aerial vehicles (UAVs) and UAV-based search system. Moving IRSTD still remains highly challenging due to weak target features and complex background interference. Accurate spatio-temporal feature modeling is crucial for moving target detection, typically achieved through either temporal differences or spatio-temporal (3D) convolutions. Temporal difference can explicitly leverage motion cues but exhibits limited capability in extracting spatial features, whereas 3D convolution effectively represents spatio-temporal features yet lacks explicit awareness of motion dynamics along the temporal dimension. In this paper, we propose a novel moving IRSTD network (TDCNet), which effectively extracts and enhances spatio-temporal features for accurate target detection. Specifically, we introduce a novel temporal difference convolution (TDC) re-parameterization module that comprises three parallel TDC blocks designed to capture contextual dependencies across different temporal ranges. Each TDC block fuses temporal difference and 3D convolution into a unified spatio-temporal convolution representation. This re-parameterized module can effectively capture multi-scale motion contextual features while suppressing pseudo-motion clutter in complex backgrounds, significantly improving detection performance. Moreover, we propose a TDC-guided spatio-temporal attention mechanism that performs cross-attention between the spatio-temporal features from the TDC-based backbone and a parallel 3D backbone. This mechanism models their global semantic dependencies to refine the current frame's features. Extensive experiments on IRSTD-UAV and public infrared datasets demonstrate that our TDCNet achieves state-of-the-art detection performance in moving target detection.

Figures (13)

• Figure 1: Three representative categories of methods for moving infrared small target detection. (a) Single-frame methods 2024CVPRLiuMSHNet employ 2D convolution, which lacks temporal context and often fails to distinguish targets from background clutter. (b) Multi-frame methods 2025EAAIPengSTME typically utilize 3D convolution to extract spatio-temporal features, but they often overlook explicitly leveraging motion cues, resulting in limited detection performance. (c) Our method introduces temporal difference convolution (TDC) to explicitly capture motion-contextual information while representing spatio-temporal features, thereby effectively suppressing complex backgrounds and enhancing the detection performance of moving infrared small targets.
• Figure 1: (a) Sequential combination of existing temporal difference and 3D convolution, and (b) Our proposed short-term temporal difference convolution (S-TDC) block, which fuses temporal difference modeling and 3D convolution into a unified spatio-temporal convolution representation.
• Figure 2: Overview of the proposed TDCNet. The input consists of a frame sequence $\{X_i\}_{i=1}^T$ and the current frame $X_T$. The temporal difference convolution (TDC) backbone utilizes the temporal difference convolution re-parameterization layer to extract TDC features from $\{X_i\}_{i=1}^T$. The 2D backbone processes $X_T$ to extract spatial features, while the 3D backbone handles $\{X_i\}_{i=1}^T$ to extract spatio-temporal features. The TDC-guided spatio-temporal attention module refines these feature streams to generate spatio-temporal enhanced features, which are aggregated by the neck and detection head to produce the final detection result.
• Figure 2: (a) Sequential combination of existing temporal difference and 3D convolution, and (b) Our proposed mid-term temporal difference convolution (M-TDC) block, which fuses temporal difference modeling and 3D convolution into a unified spatio-temporal convolution representation.
• Figure 3: Overview of the proposed temporal difference convolution re-parameterization (TDCR) module, which equivalently transforms three parallel TDC blocks (a) into a single 3D convolution representation (b).