RS-Mamba for Large Remote Sensing Image Dense Prediction

TL;DR

The paper tackles the need for efficient, global-context modeling in very-high-resolution remote sensing dense prediction. It introduces Remote Sensing Mamba (RSM), built on State Space Models with linear complexity, and an Omnidirectional Selective Scan Module to capture large spatial features across multiple directions without patching the entire image. Through two lightweight architectures, RSM-SS for semantic segmentation and RSM-CD for change detection, the approach achieves state-of-the-art results on multiple datasets while maintaining efficiency. The findings highlight the potential of SSM-based methods as strong, scalable baselines for processing large RS imagery, surpassing patch-based transformer approaches in both speed and accuracy.

Abstract

Context modeling is critical for remote sensing image dense prediction tasks. Nowadays, the growing size of very-high-resolution (VHR) remote sensing images poses challenges in effectively modeling context. While transformer-based models possess global modeling capabilities, they encounter computational challenges when applied to large VHR images due to their quadratic complexity. The conventional practice of cropping large images into smaller patches results in a notable loss of contextual information. To address these issues, we propose the Remote Sensing Mamba (RSM) for dense prediction tasks in large VHR remote sensing images. RSM is specifically designed to capture the global context of remote sensing images with linear complexity, facilitating the effective processing of large VHR images. Considering that the land covers in remote sensing images are distributed in arbitrary spatial directions due to characteristics of remote sensing over-head imaging, the RSM incorporates an omnidirectional selective scan module to globally model the context of images in multiple directions, capturing large spatial features from various directions. Extensive experiments on semantic segmentation and change detection tasks across various land covers demonstrate the effectiveness of the proposed RSM. We designed simple yet effective models based on RSM, achieving state-of-the-art performance on dense prediction tasks in VHR remote sensing images without fancy training strategies. Leveraging the linear complexity and global modeling capabilities, RSM achieves better efficiency and accuracy than transformer-based models on large remote sensing images. Interestingly, we also demonstrated that our model generally performs better with a larger image size on dense prediction tasks. Our code is available at https://github.com/walking-shadow/Official_Remote_Sensing_Mamba.

Figures (8)

• Figure 1: Illustration of the large image preprocessing strategy of transformer-based models. Dividing a large VHR remote sensing image into small patches results in the loss of many spatial features. Each patch contains very limited contextual information compared to the original large image.
• Figure 2: Illustration of the Overall structure of RSM-SS and RSM-CD. RSM-SS and RSM-CD can globally model the context of images in multiple directions with linear complexity using the omnidirectional selective scan.
• Figure 3: Illustration of the selective scan directions of Vim, VMamba, and OSSM.
• Figure 4: Illustration of the structure of the omnidirectional selective scan module.
• Figure 5: Sample inference results of RSM-SS on the semantic segmentation task. The results on the Massachusetts Road and WHU datasets are shown in the first and second rows, respectively. Red areas denote false positives and blue areas denote false negatives. (a) Input image. (b) Ground truth image. (c) RSM-SS result.