Enhancing Evaluation Methods for Infrared Small-Target Detection in Real-world Scenarios

TL;DR

This work targets the gap between conventional evaluation metrics and real-world performance in infrared small target detection (IRSTD) by analyzing the pivotal role of thresholding. It argues that global thresholding, coupled with metrics tailored to practical detection, yields more meaningful assessments than local thresholding or traditional ROC-based post-thresholding measures. The authors introduce a unified post-thresholding evaluation workflow centered on a maximum threshold parameter $k_{\max}$ and a corresponding $P_{fa,\min}$, alongside adapting pre-thresholding metrics like $SCR$ and $BSF$ for alignment with real systems. They validate the approach on five widely used IRSTD algorithms, finding that $ADMD$ and $NIPPS$ most consistently enhance target regions while controlling false alarms, with the new metrics aligning with qualitative observations. The proposed framework offers a practical, threshold-aware benchmark for IRSTD research and deployment.

Abstract

Infrared small target detection (IRSTD) poses a significant challenge in the field of computer vision. While substantial efforts have been made over the past two decades to improve the detection capabilities of IRSTD algorithms, there has been a lack of extensive investigation into the evaluation metrics used for assessing their performance. In this paper, we employ a systematic approach to address this issue by first evaluating the effectiveness of existing metrics and then proposing new metrics to overcome the limitations of conventional ones. To achieve this, we carefully analyze the necessary conditions for successful detection and identify the shortcomings of current evaluation metrics, including both pre-thresholding and post-thresholding metrics. We then introduce new metrics that are designed to align with the requirements of real-world systems. Furthermore, we utilize these newly proposed metrics to compare and evaluate the performance of five widely recognized small infrared target detection algorithms. The results demonstrate that the new metrics provide consistent and meaningful quantitative assessments, aligning with qualitative observations.

Figures (17)

• Figure 1: The block diagram of a typical IRSTD pipline
• Figure 2: Variable dynamic range in saliency map. a) Original infrared image. b) filtering result using TopHat algorithm zhu2020balanced, c) filtering result using AAGD moradi2018false algorithm. The dynamic range of the saliency map might be different than input infrared image depending on the applied IRSTD.
• Figure 3: The automatic thresholding results. a) Original infrared image. b) Top-hat filtering result zhu2020balanced. c) Otsu's thresholding result $(T=0.48)$. d) automatic thresholding using average values of background and target classes $(T=19)$. e) Manual thresholding $(T=29)$.
• Figure 4: The automatic thresholding results. a) Original noisy infrared image. b) Top-hat filtering result. c) Otsu's thresholding result $(T=0.5)$. d) automatic thresholding using average values of background and target classes $(T=7)$. e) Manual thresholding $(T=10)$.
• Figure 5: Drawback of automatic thresholding in scenarios with no targets. a) original infrared image which does not contain small target. b) the result of Top-Hat filtering. c) automatic thresholding results.