Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

X-VMamba: Explainable Vision Mamba

Mohamed A. Mabrok, Yalda Zafari

TL;DR

This paper tackles the interpretability gap for Vision State Space Models by introducing a principled controllability-based framework to quantify how input elements steer the internal state dynamics and final outputs. It presents two complementary formulations—a Jacobian-based method applicable to arbitrary SSMs and a Gramian-based method for diagonal SSMs—each offering linear-time, single-pass analysis without architectural changes. Empirical validation on three medical imaging modalities with the MedMamba model shows hierarchical refinement: early layers capture diffuse textures while deeper layers focus on diagnostically meaningful patterns, with controllability signatures aligning to clinical criteria and being modulated by scanning strategies. The framework positions controllability analysis as a unified, cross-domain interpretability paradigm for SSMs, with broad potential for computer vision, language processing, and multimodal tasks, and suggests future work on non-linear dynamics, end-to-end attribution, and temporal causality.

Abstract

State Space Models (SSMs), particularly the Mamba architecture, have recently emerged as powerful alternatives to Transformers for sequence modeling, offering linear computational complexity while achieving competitive performance. Yet, despite their effectiveness, understanding how these Vision SSMs process spatial information remains challenging due to the lack of transparent, attention-like mechanisms. To address this gap, we introduce a controllability-based interpretability framework that quantifies how different parts of the input sequence (tokens or patches) influence the internal state dynamics of SSMs. We propose two complementary formulations: a Jacobian-based method applicable to any SSM architecture that measures influence through the full chain of state propagation, and a Gramian-based approach for diagonal SSMs that achieves superior speed through closed-form analytical solutions. Both methods operate in a single forward pass with linear complexity, requiring no architectural modifications or hyperparameter tuning. We validate our framework through experiments on three diverse medical imaging modalities, demonstrating that SSMs naturally implement hierarchical feature refinement from diffuse low-level textures in early layers to focused, clinically meaningful patterns in deeper layers. Our analysis reveals domain-specific controllability signatures aligned with diagnostic criteria, progressive spatial selectivity across the network hierarchy, and the substantial influence of scanning strategies on attention patterns. Beyond medical imaging, we articulate applications spanning computer vision, natural language processing, and cross-domain tasks. Our framework establishes controllability analysis as a unified, foundational interpretability paradigm for SSMs across all domains. Code and analysis tools will be made available upon publication

X-VMamba: Explainable Vision Mamba

TL;DR

This paper tackles the interpretability gap for Vision State Space Models by introducing a principled controllability-based framework to quantify how input elements steer the internal state dynamics and final outputs. It presents two complementary formulations—a Jacobian-based method applicable to arbitrary SSMs and a Gramian-based method for diagonal SSMs—each offering linear-time, single-pass analysis without architectural changes. Empirical validation on three medical imaging modalities with the MedMamba model shows hierarchical refinement: early layers capture diffuse textures while deeper layers focus on diagnostically meaningful patterns, with controllability signatures aligning to clinical criteria and being modulated by scanning strategies. The framework positions controllability analysis as a unified, cross-domain interpretability paradigm for SSMs, with broad potential for computer vision, language processing, and multimodal tasks, and suggests future work on non-linear dynamics, end-to-end attribution, and temporal causality.

Abstract

State Space Models (SSMs), particularly the Mamba architecture, have recently emerged as powerful alternatives to Transformers for sequence modeling, offering linear computational complexity while achieving competitive performance. Yet, despite their effectiveness, understanding how these Vision SSMs process spatial information remains challenging due to the lack of transparent, attention-like mechanisms. To address this gap, we introduce a controllability-based interpretability framework that quantifies how different parts of the input sequence (tokens or patches) influence the internal state dynamics of SSMs. We propose two complementary formulations: a Jacobian-based method applicable to any SSM architecture that measures influence through the full chain of state propagation, and a Gramian-based approach for diagonal SSMs that achieves superior speed through closed-form analytical solutions. Both methods operate in a single forward pass with linear complexity, requiring no architectural modifications or hyperparameter tuning. We validate our framework through experiments on three diverse medical imaging modalities, demonstrating that SSMs naturally implement hierarchical feature refinement from diffuse low-level textures in early layers to focused, clinically meaningful patterns in deeper layers. Our analysis reveals domain-specific controllability signatures aligned with diagnostic criteria, progressive spatial selectivity across the network hierarchy, and the substantial influence of scanning strategies on attention patterns. Beyond medical imaging, we articulate applications spanning computer vision, natural language processing, and cross-domain tasks. Our framework establishes controllability analysis as a unified, foundational interpretability paradigm for SSMs across all domains. Code and analysis tools will be made available upon publication

Paper Structure

This paper contains 21 sections, 22 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. State Space Models (SSMs)
  3. Explainability Foundation for SSMs
  4. Initial Formulation: A Direct Application of Controllability Theory
  5. Influence on the Aggregated Output (Jacobian Method)
  6. Gramian-Based Influence Score (Gramian Method )
  7. Experimental Evaluation on Medical Image Classification
  8. Datasets and Model
  9. Results
  10. Potential Applications
  11. Computer Vision Applications
  12. Video Understanding and Temporal Reasoning
  13. Autonomous Driving and Safety-Critical Perception
  14. Adversarial Robustness and Model Security
  15. Remote Sensing and Large-Scale Spatial Analysis
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Heatmap of classification SSM model using several datasets in different architectural layers.
  • Figure 2: An illustration of influence score distribution across different layers of the model.
  • Figure 3: An illustration of influence score distribution across different image scanning schemes within different layers of the model.