Selective Structured State Space for Multispectral-fused Small Target Detection

TL;DR

The paper tackles small-target detection in high-resolution multispectral remote sensing by proposing the $S_{6}^{4}$-MSTD framework, which leverages the linear-time Mamba backbone augmented with ESTD and CARG to improve local and global feature representation for small targets. It introduces the lightweight Mask Enhanced Pixel-level Fusion (MEPF) module to fuse visible and infrared data efficiently, achieving high-quality fusion with about 1650 parameters. Ablation studies show additive gains from MEPF, ESTD, and CARG, while experiments on DroneVehicle and VEDAI datasets demonstrate strong small-target detection performance and favorable speed-accuracy trade-offs suitable for edge devices. Overall, the approach delivers competitive accuracy with significantly lower computational burden compared to existing multimodal and Transformer-based methods, enabling real-time multispectral small-target detection in remote sensing applications.

Abstract

Target detection in high-resolution remote sensing imagery faces challenges due to the low recognition accuracy of small targets and high computational costs. The computational complexity of the Transformer architecture increases quadratically with image resolution, while Convolutional Neural Networks (CNN) architectures are forced to stack deeper convolutional layers to expand their receptive fields, leading to an explosive growth in computational demands. To address these computational constraints, we leverage Mamba's linear complexity for efficiency. However, Mamba's performance declines for small targets, primarily because small targets occupy a limited area in the image and have limited semantic information. Accurate identification of these small targets necessitates not only Mamba's global attention capabilities but also the precise capture of fine local details. To this end, we enhance Mamba by developing the Enhanced Small Target Detection (ESTD) module and the Convolutional Attention Residual Gate (CARG) module. The ESTD module bolsters local attention to capture fine-grained details, while the CARG module, built upon Mamba, emphasizes spatial and channel-wise information, collectively improving the model's ability to capture distinctive representations of small targets. Additionally, to highlight the semantic representation of small targets, we design a Mask Enhanced Pixel-level Fusion (MEPF) module for multispectral fusion, which enhances target features by effectively fusing visible and infrared multimodal information.

Figures (11)

• Figure 1: The comparison of previous multispectral fusion detection methods (a) and Ours (b).
• Figure 2: The comparison of Vmamba (a) and Our method (b). And structure of SS2D (c). Four different scan paths are set for traversing, and each sequence is processed independently.
• Figure 3: An overview of $S_{6}^{4}$-MSTD. It consists of four components which are the Mask Enhanced Pixel-level Fusion Module, the Enhanced Small Target Detection State Space Backbone, the YOLOv8n Head, and the Detection Network. The captured visible and infrared images are firstly fused by the MEPF module to generate the fused images $I_f$. The task of the backbone module is to extract features from the fused images. We introduce the ESTD module and CARG module to enhance Mamba's feature extraction capability specifically for small targets. After this, the head network fuses the features extracted at different stages. Finally, it is fed into the detection network to get the detection results.
• Figure 4: The architecture of the mask enhanced pixel-level fusion module consists of a split module, a mask generation module, and a feature fusion module, where $x_{in}$ is the input, $I_f$ is the pixel-level fused image, and $k=3$ indicates a convolutional kernel size of 3$\times$3. We will detail the implementation in \ref{['sec:mepf']}.
• Figure 5: Structure of ESTVSS Block and VisionClueMerge. MEPF module generates $I_{f}$. After that, we use ESTVSS Block for feature extraction. VisionClueMerge is used to merge and stitch tensors. The ESTD module and the CARG module are used to enhance the detection of small targets.