Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BadVim: Unveiling Backdoor Threats in Visual State Space Model

Cheng-Yi Lee, Yu-Hsuan Chiang, Zhong-You Wu, Chia-Mu Yu, Chun-Shien Lu

TL;DR

This work analyzes the vulnerability of Visual State Space Models (VSSMs) to backdoor attacks and introduces BadVim, a backdoor framework that poisons training data at a rate of $0.3\%$ using low-rank perturbation triggers embedded across every patch. BadVim achieves high attack success, with ASR reaching up to $97.9\%$ on ImageNet, while preserving clean accuracy to within $\leq 1\%$, and often evades existing defenses. Across CIFAR-10, GTSRB, and ImageNet, BadVim demonstrates that the VSSM's state-transition dynamics, rooted in the NPLR/S4 framework, can be exploited to produce robust, transferable backdoors that are hard to detect via heatmaps or patch-based defenses. The results suggest that the enhanced expressivity and long-range reasoning of VSSMs come with notable security costs, motivating future work on defenses that specifically address backdoors in state-space representations.

Abstract

Visual State Space Models (VSSM) have shown remarkable performance in various computer vision tasks. However, backdoor attacks pose significant security challenges, causing compromised models to predict target labels when specific triggers are present while maintaining normal behavior on benign samples. In this paper, we investigate the robustness of VSSMs against backdoor attacks. Specifically, we delicately design a novel framework for VSSMs, dubbed BadVim, which utilizes low-rank perturbations on state-wise to uncover their impact on state transitions during training. By poisoning only $0.3\%$ of the training data, our attacks cause any trigger-embedded input to be misclassified to the targeted class with a high attack success rate (over 97%) at inference time. Our findings suggest that the state-space representation property of VSSMs, which enhances model capability, may also contribute to its vulnerability to backdoor attacks. Our attack exhibits effectiveness across three datasets, even bypassing state-of-the-art defenses against such attacks. Extensive experiments show that the backdoor robustness of VSSMs is comparable to that of Transformers (ViTs) and superior to that of Convolutional Neural Networks (CNNs). We believe our findings will prompt the community to reconsider the trade-offs between performance and robustness in model design.

BadVim: Unveiling Backdoor Threats in Visual State Space Model

TL;DR

This work analyzes the vulnerability of Visual State Space Models (VSSMs) to backdoor attacks and introduces BadVim, a backdoor framework that poisons training data at a rate of using low-rank perturbation triggers embedded across every patch. BadVim achieves high attack success, with ASR reaching up to on ImageNet, while preserving clean accuracy to within , and often evades existing defenses. Across CIFAR-10, GTSRB, and ImageNet, BadVim demonstrates that the VSSM's state-transition dynamics, rooted in the NPLR/S4 framework, can be exploited to produce robust, transferable backdoors that are hard to detect via heatmaps or patch-based defenses. The results suggest that the enhanced expressivity and long-range reasoning of VSSMs come with notable security costs, motivating future work on defenses that specifically address backdoors in state-space representations.

Abstract

Visual State Space Models (VSSM) have shown remarkable performance in various computer vision tasks. However, backdoor attacks pose significant security challenges, causing compromised models to predict target labels when specific triggers are present while maintaining normal behavior on benign samples. In this paper, we investigate the robustness of VSSMs against backdoor attacks. Specifically, we delicately design a novel framework for VSSMs, dubbed BadVim, which utilizes low-rank perturbations on state-wise to uncover their impact on state transitions during training. By poisoning only of the training data, our attacks cause any trigger-embedded input to be misclassified to the targeted class with a high attack success rate (over 97%) at inference time. Our findings suggest that the state-space representation property of VSSMs, which enhances model capability, may also contribute to its vulnerability to backdoor attacks. Our attack exhibits effectiveness across three datasets, even bypassing state-of-the-art defenses against such attacks. Extensive experiments show that the backdoor robustness of VSSMs is comparable to that of Transformers (ViTs) and superior to that of Convolutional Neural Networks (CNNs). We believe our findings will prompt the community to reconsider the trade-offs between performance and robustness in model design.
Paper Structure (32 sections, 2 theorems, 18 equations, 8 figures, 9 tables, 1 algorithm)

This paper contains 32 sections, 2 theorems, 18 equations, 8 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Related Works
  2. Visual State Space Model
  3. Backdoor Attack and Defense
  4. Backdoor Robustness of Generic Vision Models
  5. Preliminary
  6. State Space Model
  7. Formulation of VSSMs
  8. The Proposed BadVim Framework
  9. Threat Model
  10. Intuition Behind our Backdoor Attack
  11. Our Low-Rank Perturbation Triggers
  12. Why BadVim Works: Frequency Sensitivity in SSMs
  13. Experiments
  14. Evaluation Settings
  15. Empirical Results
  16. ...and 17 more sections

Key Result

Proposition 1

Consider the discrete state-space model: $h_t = \mathbf{\overline{A}}h_{t-1} + \mathbf{\overline{B}}x_t, \quad y_t = \mathbf{C}h_t$. Suppose that the input is corrupted by a low-rank perturbation $\delta$, i.e., $\hat{x}_t= x_t + \delta$. Let $\mathbf{\overline{A}'}$ denote the effective state trans

Figures (8)

  • Figure 1: Overview of BadVim framework. Poisoned samples are crafted by injecting low-rank perturbations $\delta$ into each patch state. An honest user downloads the poisoned data and trains a model, which malfunctions when a trigger is present during inference. Notably, the hidden attention matrices ali2024hidden of poisoned samples remain similar to those of clean samples.
  • Figure 2: Illustration of the three strategies in BadVim. Here, the trigger is magnified 20 times for clarity.
  • Figure 3: Visualization of VSSM under implicit attention matrices ali2024hidden on ImageNet. We evaluate clean samples using the clean model (second column) and poisoned samples using the backdoored model (third to fifth columns), respectively.
  • Figure 4: Effect of poisoning rate.
  • Figure 5: Effect of blending rate.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Proposition 1: Perturbation Persistence
  • Theorem 2
  • proof : Proof of Proposition 1