Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection

TL;DR

MS2Fusion introduces a novel multispectral feature fusion framework based on state-space modeling to jointly capture cross-modal complementary and shared information. It assembles three SSM-based modules—CP-SSM for cross-parameter complementarity, SP-SSM for shared-parameter alignment, and FF-SSM for bidirectional fusion that expands the Effective Receptive Field—into a unified MS2Fusion architecture. By operating on multi-scale features from RGB and thermal streams, MS2Fusion delivers state-of-the-art results on RGB-T object detection benchmarks (FLIR, LLVIP, M$^3$FD, VEDAI) and generalizes to RGB-T semantic segmentation and salient object detection, often with lower computational cost than Transformer-based rivals. The work demonstrates robust performance under varying modalities and input conditions, with extensive ablations validating the complementary and shared feature learning, fusion layer choices, and input configurations. Overall, MS2Fusion provides a practical, scalable solution for multispectral perception that preserves detailed local cues while leveraging global context, enabling real-time, cross-modal detection and perception in challenging environments.

Abstract

Modern multispectral feature fusion for object detection faces two critical limitations: (1) Excessive preference for local complementary features over cross-modal shared semantics adversely affects generalization performance; and (2) The trade-off between the receptive field size and computational complexity present critical bottlenecks for scalable feature modeling. Addressing these issues, a novel Multispectral State-Space Feature Fusion framework, dubbed MS2Fusion, is proposed based on the state space model (SSM), achieving efficient and effective fusion through a dual-path parametric interaction mechanism. More specifically, the first cross-parameter interaction branch inherits the advantage of cross-attention in mining complementary information with cross-modal hidden state decoding in SSM. The second shared-parameter branch explores cross-modal alignment with joint embedding to obtain cross-modal similar semantic features and structures through parameter sharing in SSM. Finally, these two paths are jointly optimized with SSM for fusing multispectral features in a unified framework, allowing our MS2Fusion to enjoy both functional complementarity and shared semantic space. In our extensive experiments on mainstream benchmarks including FLIR, M3FD and LLVIP, our MS2Fusion significantly outperforms other state-of-the-art multispectral object detection methods, evidencing its superiority. Moreover, MS2Fusion is general and applicable to other multispectral perception tasks. We show that, even without specific design, MS2Fusion achieves state-of-the-art results on RGB-T semantic segmentation and RGBT salient object detection, showing its generality. The source code will be available at https://github.com/61s61min/MS2Fusion.git.

Figures (15)

• Figure 1: The pros and cons of RGB (left) and thermal (right) images. (a) Both modalities provide complementary information, and their fusion enables more robust object detection; (b) Dual-modal shared features become crucial, since neither modality stands out distinctly. (e.g., modality-specific characteristics such as texture and thermal radiation are blurred, and cross-modal consistent features like object contours and structures are helpful for detection.)
• Figure 2: Comparing Transformer-based fusion (a), Mamba-based fusion (b) and our proposed MS2Fusion (c), with $F_T$ and $F_V$ as the input thermal and RGB image features, respectively. In Transformer-based method (a), $F_T$ and $F_V$ are fused through the multi-head attention mechanism, effectively integrating complementary information and enhancing performance across scenarios. The traditional Mamba approach (b) directly mixes dual-modal features to generate B, C, and $\Delta$ parameters for SSM-based feature interaction, which may lead to modal misalignment and feature redundancy. In contrast, our method (c) first performs intra-modal feature interaction and then extracts cross-modal shared features, achieving better modal alignment and fusion, thereby providing more robust and unified feature representations.
• Figure 3: Effective receptive field visualizations comparing CNN-based fusion method (a), Transformer-based fusion method (b), and the proposed MS2Fusion (c) method. Quantitative analysis demonstrates that MS2Fusion achieves significantly broader receptive field coverage compared to the others.
• Figure 4: Details of SSM, where the red, green and black lines correspond to the equations of \ref{['dis_SSM']}, respectively. (For an $L \times d$ dimensional input {$x_1, x_2, ...,x_L$}.) The input sequence is linearly projected to generate parameters for the state equation, followed by recursive computation via matrices A, B, and C, with an optional skip connection D. The output sequence retains the same dimensionality as the input.
• Figure 5: Overview of the model architecture. It consists of three main stages: (1) feature extraction with two backbone networks; (2) cross-modal feature fusion of $P3$, $P4$ and $P5$ with MS2Fusion module; (3) the detection results are generated through the Neck and Head layers. In our experiments, two distinct detection heads ( CoDetr and YOLOv5) are evaluated independently. The MS2Fusion module employs a dual-branch architecture to process the features of two modalities, $F_V$ and $F_T$. CP-SSM learns cross-modal complementary features, while SP-SSM extracts shared features. Finally, FF-SSM performs single brach feature enhancement and cross-modal fusion, outputting the fused feature $F_{fused}$.