OCTOPUS: Enhancing the Spatial-Awareness of Vision SSMs with Multi-Dimensional Scans and Traversal Selection

TL;DR

OCTOPUS introduces an eight-direction discrete scanning framework for Vision State-Space Models (VSSMs) to restore isotropic spatial propagation in 2D images while preserving linear complexity. By partitioning the image into independent scan-lines per direction (O-Scan), normalizing directional transitions (Normalized Transition Kernel), and fusing directional evidence with a Traversal Selection mechanism (O-Attention) and O-Merge, the method achieves improved segmentation accuracy and robust boundary delineation with competitive classification performance. Empirical results on ADE20K and miniImageNet demonstrate significant gains in dense prediction tasks and strong, consistent performance in classification, underscoring the value of geometry-aligned, multi-directional recurrence for scalable vision models.

Abstract

State space models (SSMs) have recently emerged as an alternative to transformers due to their unique ability of modeling global relationships in text with linear complexity. However, their success in vision tasks has been limited due to their causal formulation, which is suitable for sequential text but detrimental in the spatial domain where causality breaks the inherent spatial relationships among pixels or patches. As a result, standard SSMs fail to capture local spatial coherence, often linking non-adjacent patches while ignoring neighboring ones that are visually correlated. To address these limitations, we introduce OCTOPUS , a novel architecture that preserves both global context and local spatial structure within images, while maintaining the linear complexity of SSMs. OCTOPUS performs discrete reoccurrence along eight principal orientations, going forward or backward in the horizontal, vertical, and diagonal directions, allowing effective information exchange across all spatially connected regions while maintaining independence among unrelated patches. This design enables multi-directional recurrence, capturing both global context and local spatial structure with SSM-level efficiency. In our classification and segmentation benchmarks, OCTOPUS demonstrates notable improvements in boundary preservation and region consistency, as evident from the segmentation results, while maintaining relatively better classification accuracy compared to existing V-SSM based models. These results suggest that OCTOPUS appears as a foundation method for multi-directional recurrence as a scalable and effective mechanism for building spatially aware and computationally efficient vision architectures.

Figures (9)

• Figure 1: Comparing directional scanning in VMamba and Octopus.(a) VMamba: Uses continuous row-wise scans in four directions ($\rightarrow, \leftarrow, \downarrow, \uparrow$). However, due to the continuity between rows, non-adjacent patches (e.g., the red regions across rows) become connected despite lacking spatial proximity — leading to no spatial context. Moreover, patches at actual spatial boundaries (from green to red) are not connected, creating a spatial discrepancy. (b) Octopus: Employs discrete, independent scans across eight spatial directions (horizontal, vertical, and diagonal). This preserves spatial order, ensuring that only spatially adjacent patches interact while maintaining complete directional recurrence across all neighborhoods.
• Figure 2: Overview of our pipeline. O-Scan performs eight discrete canonical traversals, $\{\mathrm{row}_{\!\rightarrow},\; \mathrm{row}_{\!\leftarrow},\; \mathrm{col}_{\!\downarrow},\; \mathrm{col}_{\!\uparrow},\; \mathrm{diag}_{\!\searrow},\; \mathrm{diag}_{\!\nwarrow},\; \mathrm{diag}_{\!\swarrow},\; \mathrm{diag}_{\!\nearrow}\}$, covering the complete set of horizontal, vertical, and diagonal orientations. The | markers in O-Scan denote boundaries between independent scan-lines, where the hidden state is reset, forming the core of our discrete scan formulation. Each directional sequence is processed by the Selective Scan Block, fused via O-Merge, and aggregated with the Traversal Selection Mechanism to produce the final output patches.
• Figure 3: Overall Architecture of Octopus: (a) The full hierarchical encoder consisting of four stages. Each stage applies multiple O-VSS blocks, with spatial resolution preserved inside the stage and reduced via downsampling between stages. (b) The O-VSS block design, composed of Layer Normalization, an O-SS2D block (which integrates the proposed 8-direction Selective Scan), and an FFN. The O-SS2D block itself combines O-Scan operations with an O-Merge module to fuse multi-directional features.
• Figure 4: Ablation on directional scans and traversal selection.Left: Increasing the number of canonical scan directions from 2 to 8 steadily improves recognition performance, validating the effectiveness of our discrete multi-line scan formulation. Right: Incorporating the Traversal Selection Mechanism (TSM) further enhances Top-1 accuracy by enabling data-dependent fusion of directional evidence.
• Figure 5: ERF evolution across models Comparison of effective receptive fields (ERFs) before and after training across different architectures. Octopus shows the most structured and spatially aware ERF pattern.