Rethinking Infrared Small Target Detection: A Foundation-Driven Efficient Paradigm

TL;DR

This work tackles infrared small target detection (SIRST) by leveraging frozen representations from Visual Foundation Models within a Foundation-Driven Efficient Paradigm (FDEP). It introduces a Semantic Alignment Modulation Fusion (SAMF) module to fuse global semantic priors with task-specific features, and a Collaborative Optimization-based Implicit Self-Distillation (CO-ISD) strategy to enable semantic transfer without inflating inference costs. To enable fair, stable comparisons, the authors propose the Holistic SIRST Evaluation (HSE) metric that jointly assesses pixel-level confidence and target-level robustness. Empirical results show SIRST detectors equipped with FDEP achieve state-of-the-art performance across multiple public datasets while maintaining efficient inference, demonstrating strong generalization and deployability. The work highlights a practical pathway for integrating foundation-model knowledge into specialized infrared perception tasks.

Abstract

While large-scale visual foundation models (VFMs) exhibit strong generalization across diverse visual domains, their potential for single-frame infrared small target (SIRST) detection remains largely unexplored. To fill this gap, we systematically introduce the frozen representations from VFMs into the SIRST task for the first time and propose a Foundation-Driven Efficient Paradigm (FDEP), which can seamlessly adapt to existing encoder-decoder-based methods and significantly improve accuracy without additional inference overhead. Specifically, a Semantic Alignment Modulation Fusion (SAMF) module is designed to achieve dynamic alignment and deep fusion of the global semantic priors from VFMs with task-specific features. Meanwhile, to avoid the inference time burden introduced by VFMs, we propose a Collaborative Optimization-based Implicit Self-Distillation (CO-ISD) strategy, which enables implicit semantic transfer between the main and lightweight branches through parameter sharing and synchronized backpropagation. In addition, to unify the fragmented evaluation system, we construct a Holistic SIRST Evaluation (HSE) metric that performs multi-threshold integral evaluation at both pixel-level confidence and target-level robustness, providing a stable and comprehensive basis for fair model comparison. Extensive experiments demonstrate that the SIRST detection networks equipped with our FDEP framework achieve state-of-the-art (SOTA) performance on multiple public datasets. Our code is available at https://github.com/YuChuang1205/FDEP-Framework

Figures (7)

• Figure 1: Comparison of different SIRST detection paradigms. Top: Traditional encoder-decoder-based architecture performs feature encoding and decoding in a sequential manner. Bottom: Our FDEP framework consists of a main branch (purple arrow) and a lightweight branch (green arrow). The main branch integrates frozen semantic representations from VFMs. The lightweight branch maintains the original structure. During inference, only the lightweight branch is used for efficiency.
• Figure 2: Overview of the FDEP Framework.
• Figure 3: Visualization of several excellent methods on the SIRST3 dataset. Red: correct detections; blue: false positives; orange: missed detections. Every two rows from top to bottom: Image, “Original”, “+ FDEP w/o CO-ISD”, “+ FDEP Framework”, True label.
• Figure 4: ROC curves for different cases. Red represents the High-AUC case. Blue represents the Low-AUC case. The three columns on the left denote the original image, the true label, and the simulated prediction results, respectively.
• Figure S1: Visualization of several excellent methods on the NUAA-SIRST dataset. Red: correct detections; blue: false positives; orange: missed detections. Every two rows from top to bottom: Image, "Original", "+ FDEP w/o CO-ISD", "+ FDEP Framework", True label.