Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation

Xianping Ma, Xiaokang Zhang, Man-On Pun

TL;DR

RS3Mamba introduces a dual-branch architecture for remote sensing image semantic segmentation that leverages a Visual State Space (VSS) auxiliary encoder to capture global context while maintaining CNN-like local feature extraction in the main encoder. A Collaborative Completion Module (CCM) fuses cross-branch information across multiple scales, combining a Swin-style global attention branch with local convolutions for robust feature fusion. Evaluations on ISPRS Vaihingen and LoveDA Urban datasets show consistent improvements over CNN- and Transformer-based baselines in terms of mean F1-score and mean IoU, demonstrating the effectiveness of incorporating VSS-based global information in RS tasks. The work highlights a practical, linear-complexity approach to long-range modeling in remote sensing and provides insights for broader adoption of Mamba-based methods in geoscience applications, with code to be released.

Abstract

Semantic segmentation of remote sensing images is a fundamental task in geoscience research. However, there are some significant shortcomings for the widely used convolutional neural networks (CNNs) and Transformers. The former is limited by its insufficient long-range modeling capabilities, while the latter is hampered by its computational complexity. Recently, a novel visual state space (VSS) model represented by Mamba has emerged, capable of modeling long-range relationships with linear computability. In this work, we propose a novel dual-branch network named remote sensing images semantic segmentation Mamba (RS3Mamba) to incorporate this innovative technology into remote sensing tasks. Specifically, RS3Mamba utilizes VSS blocks to construct an auxiliary branch, providing additional global information to convolution-based main branch. Moreover, considering the distinct characteristics of the two branches, we introduce a collaborative completion module (CCM) to enhance and fuse features from the dual-encoder. Experimental results on two widely used datasets, ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness and potential of the proposed RS3Mamba. To the best of our knowledge, this is the first vision Mamba specifically designed for remote sensing images semantic segmentation. The source code will be made available at https://github.com/sstary/SSRS.

RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation

TL;DR

RS3Mamba introduces a dual-branch architecture for remote sensing image semantic segmentation that leverages a Visual State Space (VSS) auxiliary encoder to capture global context while maintaining CNN-like local feature extraction in the main encoder. A Collaborative Completion Module (CCM) fuses cross-branch information across multiple scales, combining a Swin-style global attention branch with local convolutions for robust feature fusion. Evaluations on ISPRS Vaihingen and LoveDA Urban datasets show consistent improvements over CNN- and Transformer-based baselines in terms of mean F1-score and mean IoU, demonstrating the effectiveness of incorporating VSS-based global information in RS tasks. The work highlights a practical, linear-complexity approach to long-range modeling in remote sensing and provides insights for broader adoption of Mamba-based methods in geoscience applications, with code to be released.

Abstract

Semantic segmentation of remote sensing images is a fundamental task in geoscience research. However, there are some significant shortcomings for the widely used convolutional neural networks (CNNs) and Transformers. The former is limited by its insufficient long-range modeling capabilities, while the latter is hampered by its computational complexity. Recently, a novel visual state space (VSS) model represented by Mamba has emerged, capable of modeling long-range relationships with linear computability. In this work, we propose a novel dual-branch network named remote sensing images semantic segmentation Mamba (RS3Mamba) to incorporate this innovative technology into remote sensing tasks. Specifically, RS3Mamba utilizes VSS blocks to construct an auxiliary branch, providing additional global information to convolution-based main branch. Moreover, considering the distinct characteristics of the two branches, we introduce a collaborative completion module (CCM) to enhance and fuse features from the dual-encoder. Experimental results on two widely used datasets, ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness and potential of the proposed RS3Mamba. To the best of our knowledge, this is the first vision Mamba specifically designed for remote sensing images semantic segmentation. The source code will be made available at https://github.com/sstary/SSRS.
Paper Structure (15 sections, 1 equation, 4 figures, 4 tables)

This paper contains 15 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Auxiliary encoder
  4. Main encoder and CCM
  5. Experiments and Discussion
  6. Datasets
  7. ISPRS Vaihingen
  8. LoveDA Urban
  9. Experimental Setup
  10. Performance Comparison
  11. Performance Comparison on the Vaihingen dataset
  12. Performance Comparison on the LoveDA Urban
  13. Ablation Study
  14. Model Complexity Analysis
  15. Conclusion

Figures (4)

  • Figure 1: The overall architecture of RS3Mamba.
  • Figure 2: (a) The detailed architecture of VSS block. (b) The detailed architecture of CCM.
  • Figure 3: Qualitative performance comparisons on the ISPRS Vaihaigen with the size of $512 \times 512$. (a) NIRRG images, (b) Ground truth, (c) ABCNet, (d) TransUNet, (e) UNetformer, (f) CMTFNet and (g) the proposed RS3Mamba. We showcase two samples for each model.
  • Figure 4: Qualitative performance comparisons on the LoveDA Urban with the size of $1024 \times 1024$. (a) NIRRG images, (b) Ground truth, (c) ABCNet, (d) TransUNet, (e) UNetformer, (f) CMTFNet and (g) the proposed RS3Mamba. We showcase two samples for each model.