MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection

TL;DR

This work tackles infrared small target detection (ISTD) by addressing the efficiency-accuracy trade-off inherent in transformer-based approaches. It introduces MiM-ISTD, a pure Mamba-based encoder that nests two levels of Mamba blocks (Inner for local-word relations and Outer for global-sentence relations) within a four-stage hierarchical architecture, enabled by a convolutional stem and a two-tier patch structure (visual sentences and visual words). The model achieves state-of-the-art accuracy while dramatically improving efficiency, demonstrating up to 8× faster inference and substantial GPU memory savings on high-resolution images. These results provide a practical, scalable solution for real-time ISTD on large infrared images and offer a template for future Mamba-based vision backbones in high-resolution detection tasks.

Abstract

Recently, infrared small target detection (ISTD) has made significant progress, thanks to the development of basic models. Specifically, the models combining CNNs with transformers can successfully extract both local and global features. However, the disadvantage of the transformer is also inherited, i.e., the quadratic computational complexity to sequence length. Inspired by the recent basic model with linear complexity for long-distance modeling, Mamba, we explore the potential of this state space model for ISTD task in terms of effectiveness and efficiency in the paper. However, directly applying Mamba achieves suboptimal performances due to the insufficient harnessing of local features, which are imperative for detecting small targets. Instead, we tailor a nested structure, Mamba-in-Mamba (MiM-ISTD), for efficient ISTD. It consists of Outer and Inner Mamba blocks to adeptly capture both global and local features. Specifically, we treat the local patches as "visual sentences" and use the Outer Mamba to explore the global information. We then decompose each visual sentence into sub-patches as "visual words" and use the Inner Mamba to further explore the local information among words in the visual sentence with negligible computational costs. By aggregating the visual word and visual sentence features, our MiM-ISTD can effectively explore both global and local information. Experiments on NUAA-SIRST and IRSTD-1k show the superior accuracy and efficiency of our method. Specifically, MiM-ISTD is $8 \times$ faster than the SOTA method and reduces GPU memory usage by 62.2$\%$ when testing on $2048 \times 2048$ images, overcoming the computation and memory constraints on high-resolution infrared images.

Figures (7)

• Figure 1: (a), (b) MiM-ISTD is more computation and memory efficient than present SOTA methods, DNANet li2022dense and TCI-Former Chen_Tan_Chu_Wu_Liu_Yu_2024, in dealing with high-resolution infrared images. Specifically, MiM-ISTD is $10 \times$ faster than TCI-Former Chen_Tan_Chu_Wu_Liu_Yu_2024 and saves 62.2$\%$ GPU memory per image with a resolution of $2048 \times 2048$. (c) The overall efficiency comparison on images of resolution $512 \times 512$, where larger bubbles denote higher GPU memory usage.
• Figure 2: Illustration of the 2D Selective Scan (SS2D) on an infrared image. We commence by scanning an image using scan expanding. The four resulting feature sequences are then individually processed through the S6 block and the four output sequences are merged (scan merging) to construct the final 2D feature map.
• Figure 3: Overview of our MiM-ISTD, which mainly includes a convolutional stem, a pure Mamba-based MiM hierarchical encoder, and a plain decoder.
• Figure 4: Overview of (a) our proposed Mamba-in-Mamba (MiM) block, which contains an Inner Mamba block and an Outer Mamba block, and (b) the structure of Inner/Outer Mamba block liu2024vmamba. The Inner Mamba block is shared in the same layer. The dashed line in (a) means bypassing the Outer Mamba block.
• Figure 5: Visualization of predicted mask results. The predicted targets are amplified in the yellow boxes for clearer observation, and we highlight some inaccuracies, such as false alarms and missed detection, made by other methods.