Spectral State Space Model for Rotation-Invariant Visual Representation Learning

TL;DR

Spectral VMamba tackles the rotation sensitivity and limited cross-patch relationships of vision state-space models by leveraging graph Laplacian spectral information to order image patches. The Rotational Feature Normalizer (RFN) ensures orientation-consistent features, while the Spectral Traversal Scan and Spectral Canonicalization provide rotation-invariant, semantically meaningful patch traversal guided by Laplacian eigenvectors $u^{(j)}$. Empirically, the method outperforms VMamba and competitive ViT-based models on miniImageNet, with notable improvements in rotation robustness and segmentation performance on ADE20K, all while maintaining lower or comparable FLOPs. This approach advances robust, globally coherent visual representations suitable for efficient large-scale vision tasks.

Abstract

State Space Models (SSMs) have recently emerged as an alternative to Vision Transformers (ViTs) due to their unique ability of modeling global relationships with linear complexity. SSMs are specifically designed to capture spatially proximate relationships of image patches. However, they fail to identify relationships between conceptually related yet not adjacent patches. This limitation arises from the non-causal nature of image data, which lacks inherent directional relationships. Additionally, current vision-based SSMs are highly sensitive to transformations such as rotation. Their predefined scanning directions depend on the original image orientation, which can cause the model to produce inconsistent patch-processing sequences after rotation. To address these limitations, we introduce Spectral VMamba, a novel approach that effectively captures the global structure within an image by leveraging spectral information derived from the graph Laplacian of image patches. Through spectral decomposition, our approach encodes patch relationships independently of image orientation, achieving rotation invariance with the aid of our Rotational Feature Normalizer (RFN) module. Our experiments on classification tasks show that Spectral VMamba outperforms the leading SSM models in vision, such as VMamba, while maintaining invariance to rotations and a providing a similar runtime efficiency.

Figures (9)

• Figure 1: Effect of image rotation on patch processing in VMamba and Spectral VMamba networks. In VMamba networks (first row), patches are traversed using predefined horizontal and vertical scanning routes. As a result, rotating the image by 90° significantly changes the sequence in which patches are processed. In contrast, Spectral VMamba (second row) organizes patches using spectral information derived from a graph laplacian of the image patches. As shown in the second row, Spectral VMamba consistently processes background patches before snake patches in both the original and rotated images. The snake image in this analysis is AI-generated.
• Figure 2: The Spectral Traversal Scan (STS) architecture begins by representing image patches as a graph. The Laplacian spectrum of this graph is then generated, and its eigenvectors are computed. The traversal path for the image patches is determined by the order of eigenvectors. The 1st eigenvector traverses the foreground/background regions, the 2nd eigenvector focuses on the next most salient structures, and each subsequent eigenvector progressively captures finer details in the image.
• Figure 3: An overview of the proposed method. Our network consists of two primary components: the RFN module, which normalizes feature orientation to ensure consistency across different perspectives, and a series of Spectral VMamba blocks that process data in three stages: (1) Spectral Traversal Scan (STS), (2) S6 block (selective scan), and (3) Spectral Traversal Merge (STM). In STS, patches are reordered based on spectral coherence derived from the Laplacian spectrum. Each sequence is then processed in parallel by dedicated S6 blocks, capturing localized patterns, and finally merged in STM to reconstruct the spatial layout, producing the final feature map.
• Figure 4: Model performance comparison across various rotation angles. Our method demonstrates consistent accuracy across all rotation angles, while VMamba exhibits significant fluctuations.
• Figure 5: Model performance for different number of eigenvectors. Four eigenvectors exhibit the best performance.

Theorems & Definitions (2)

• Theorem 3.1
• proof