Sparse Deformable Mamba for Hyperspectral Image Classification

TL;DR

This work tackles the efficiency bottleneck in Mamba-based hyperspectral image classification by introducing Sparse Deformable Mamba (SDMamba). It couples Sparse Deformable Sequencing (SDS) with dedicated spatial and spectral modules (SDSpaM and SDSpeM) and an attention-based fusion to jointly learn compact, deformable token sets and fuse spatial-spectral information. Empirical results on Indian Pines and Pavia University demonstrate improved accuracy and reduced computation, with notable small-class preservation and sharper decision boundaries. Overall, the approach provides a scalable, computation-efficient pathway for high-performance HSI classification.

Abstract

Although Mamba models significantly improve hyperspectral image (HSI) classification, one critical challenge is the difficulty in building the sequence of Mamba tokens efficiently. This paper presents a Sparse Deformable Mamba (SDMamba) approach for enhanced HSI classification, with the following contributions. First, to enhance Mamba sequence, an efficient Sparse Deformable Sequencing (SDS) approach is designed to adaptively learn the ''optimal" sequence, leading to sparse and deformable Mamba sequence with increased detail preservation and decreased computations. Second, to boost spatial-spectral feature learning, based on SDS, a Sparse Deformable Spatial Mamba Module (SDSpaM) and a Sparse Deformable Spectral Mamba Module (SDSpeM) are designed for tailored modeling of the spatial information spectral information. Last, to improve the fusion of SDSpaM and SDSpeM, an attention based feature fusion approach is designed to integrate the outputs of the SDSpaM and SDSpeM. The proposed method is tested on several benchmark datasets with many state-of-the-art approaches, demonstrating that the proposed approach can achieve higher accuracy with less computation, and better detail small-class preservation capability.

Figures (5)

• Figure 1: Illustration of our proposed sparse deformable sequencing (SDS) for improving Mamba. Various classical scanning approaches in (a)-(h) are dense and predefined, because they use all tokens in a deterministic manner, causing potential redundancy and rigidity, unnecessary computation cost, and difficulty in selecting scanning approaches. In contrast, our SDS approach in (i) can identify and sequencelimited number of relevant tokens in a learnable and adaptive manner, leading to sparse and deformable sequence patterns that can reduce redundancy, rigidity and computational cost in Mamba.
• Figure 2: The proposed SDMamab is sparse, because the input sequences to the MambaBlock, i.e., $\overline{\boldsymbol{\mathit{Z}}}_j (5 \ tokens)$ and $\overline{\boldsymbol{\mathit{A}}}_j (3 \ tokens)$have much less number of tokens than $\boldsymbol{\mathit{Z}}_j (HW \ tokens)$ and $\boldsymbol{\mathit{A}}_j (C \ tokens)$ respectively. Moreover, in SDMamba, the order of tokens in $\overline{\boldsymbol{\mathit{Z}}}_j$ and $\overline{\boldsymbol{\mathit{A}}}_j$ are deformable and learnable, because two adaptive attention matrices, i.e., $SparseSpatialAttn$ and $SparseSpectralAttn$, are used to sort the tokens and identify a limited number of relevant tokens. Therefore, the proposed SDMamba approach has sparse and deformable sequence patterns that can reduce redundancy, rigidity, and computational cost in classical Mamba models.
• Figure 3: The outputs of SDSpaM and SDSpeM, i.e., $\boldsymbol{\mathit{Z}}_j$, $\boldsymbol{\mathit{A}}_j$ respectively, are fused using the attention mechanism. The fused feature map $\boldsymbol{\mathit{Y}}_j$ can better capture both spatial information and spectral information.
• Figure 4: The Indian Pines classification map generated by different methods. (a) SSRN (b) SS-ConvNeXt (c) MTGAN (d) SSFTT (e) SSTN (f) GSC-ViT (g) MammbaHSI (h) 3DSS-Mamba (i) HyperMamba (j) SDMamba (k) Ground Truth (l) RGB Image (m) TSNE features on labeled data (n) TSNE features on predicted pixel
• Figure 5: The Pavia University classification map generated by different methods. (a) SSRN (b) SS-ConvNeXt (c) MTGAN (d) SSFTT (e) SSTN (f) GSC-ViT (g) MammbaHSI (h) 3DSS-Mamba (i) HyperMamba (j) SDMamba (k) Ground Truth (l) RGB Image (m) TSNE features on labeled data (n) TSNE features on predicted pixel