Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Robustness of Visual State Space Models for Image Classification

Chengbin Du, Yanxi Li, Chang Xu

TL;DR

This paper investigates VMamba's robustness to adversarial attacks, employing both whole-image and patch-specific adversarial attacks, and explores VMamba's gradients and back-propagation during white-box attacks, uncovering unique vulnerabilities and defensive capabilities of its novel components.

Abstract

Visual State Space Model (VMamba) has recently emerged as a promising architecture, exhibiting remarkable performance in various computer vision tasks. However, its robustness has not yet been thoroughly studied. In this paper, we delve into the robustness of this architecture through comprehensive investigations from multiple perspectives. Firstly, we investigate its robustness to adversarial attacks, employing both whole-image and patch-specific adversarial attacks. Results demonstrate superior adversarial robustness compared to Transformer architectures while revealing scalability weaknesses. Secondly, the general robustness of VMamba is assessed against diverse scenarios, including natural adversarial examples, out-of-distribution data, and common corruptions. VMamba exhibits exceptional generalizability with out-of-distribution data but shows scalability weaknesses against natural adversarial examples and common corruptions. Additionally, we explore VMamba's gradients and back-propagation during white-box attacks, uncovering unique vulnerabilities and defensive capabilities of its novel components. Lastly, the sensitivity of VMamba to image structure variations is examined, highlighting vulnerabilities associated with the distribution of disturbance areas and spatial information, with increased susceptibility closer to the image center. Through these comprehensive studies, we contribute to a deeper understanding of VMamba's robustness, providing valuable insights for refining and advancing the capabilities of deep neural networks in computer vision applications.

Understanding Robustness of Visual State Space Models for Image Classification

TL;DR

This paper investigates VMamba's robustness to adversarial attacks, employing both whole-image and patch-specific adversarial attacks, and explores VMamba's gradients and back-propagation during white-box attacks, uncovering unique vulnerabilities and defensive capabilities of its novel components.

Abstract

Visual State Space Model (VMamba) has recently emerged as a promising architecture, exhibiting remarkable performance in various computer vision tasks. However, its robustness has not yet been thoroughly studied. In this paper, we delve into the robustness of this architecture through comprehensive investigations from multiple perspectives. Firstly, we investigate its robustness to adversarial attacks, employing both whole-image and patch-specific adversarial attacks. Results demonstrate superior adversarial robustness compared to Transformer architectures while revealing scalability weaknesses. Secondly, the general robustness of VMamba is assessed against diverse scenarios, including natural adversarial examples, out-of-distribution data, and common corruptions. VMamba exhibits exceptional generalizability with out-of-distribution data but shows scalability weaknesses against natural adversarial examples and common corruptions. Additionally, we explore VMamba's gradients and back-propagation during white-box attacks, uncovering unique vulnerabilities and defensive capabilities of its novel components. Lastly, the sensitivity of VMamba to image structure variations is examined, highlighting vulnerabilities associated with the distribution of disturbance areas and spatial information, with increased susceptibility closer to the image center. Through these comprehensive studies, we contribute to a deeper understanding of VMamba's robustness, providing valuable insights for refining and advancing the capabilities of deep neural networks in computer vision applications.
Paper Structure (38 sections, 8 equations, 7 figures, 9 tables)

This paper contains 38 sections, 8 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Vision Transformers
  4. State Space Models
  5. Selective State Space Models
  6. Adversarial Robustness
  7. White-box Attacks
  8. Attack the entire image.
  9. The patch-wise attack.
  10. Black-box Attacks and Transferability of Noises
  11. General Robustness
  12. Why is VMamba Robust to White-box Attacks?
  13. Sensitivity to Information Loss
  14. Sensitivity to the Dense Perturbation
  15. Sensitivity to Sparse Perturbation
  16. ...and 23 more sections

Figures (7)

  • Figure 1: An example image with its Patch-wise drop and Pixel-wise drop version.
  • Figure 2: Robust Accuracy related to the amount of information loss.
  • Figure 3: Example images and their different extend of disorder examples
  • Figure 4: Robust Accuracy related to degrees of shuffle between VMamba and Swin Model.
  • Figure 5: Performance of the VMamba-T and Swin-T models under Patch-fool attacks targeting each of 196 image patches, For the efficiency of this experiment, we randomly selected 500 images from the ImageNet-1K validation set.
  • ...and 2 more figures