Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STRAP-ViT: Segregated Tokens with Randomized -- Transformations for Defense against Adversarial Patches in ViTs

Nandish Chattopadhyay, Anadi Goyal, Chandan Karfa, Anupam Chattopadhyay

Abstract

Adversarial patches are physically realizable localized noise, which are able to hijack Vision Transformers (ViT) self-attention, pulling focus toward a small, high-contrast region and corrupting the class token to force confident misclassifications. In this paper, we claim that the tokens which correspond to the areas of the image that contain the adversarial noise, have different statistical properties when compared to the tokens which do not overlap with the adversarial perturbations. We use this insight to propose a mechanism, called STRAP-ViT, which uses Jensen-Shannon Divergence as a metric for segregating tokens that behave as anomalies in the Detection Phase, and then apply randomized composite transformations on them during the Mitigation Phase to make the adversarial noise ineffective. The minimum number of tokens to transform is a hyper-parameter for the defense mechanism and is chosen such that at least 50% of the patch is covered by the transformed tokens. STRAP-ViT fits as a non-trainable plug-and-play block within the ViT architectures, for inference purposes only, with a minimal computational cost and does not require any additional training cost/effort. STRAP-ViT has been tested on multiple pre-trained vision transformer architectures (ViT-base-16 and DinoV2) and datasets (ImageNet and CalTech-101), across multiple adversarial attacks (Adversarial Patch, LAVAN, GDPA and RP2), and found to provide excellent robust accuracies lying within a 2-3% range of the clean baselines, and outperform the state-of-the-art.

STRAP-ViT: Segregated Tokens with Randomized -- Transformations for Defense against Adversarial Patches in ViTs

Abstract

Adversarial patches are physically realizable localized noise, which are able to hijack Vision Transformers (ViT) self-attention, pulling focus toward a small, high-contrast region and corrupting the class token to force confident misclassifications. In this paper, we claim that the tokens which correspond to the areas of the image that contain the adversarial noise, have different statistical properties when compared to the tokens which do not overlap with the adversarial perturbations. We use this insight to propose a mechanism, called STRAP-ViT, which uses Jensen-Shannon Divergence as a metric for segregating tokens that behave as anomalies in the Detection Phase, and then apply randomized composite transformations on them during the Mitigation Phase to make the adversarial noise ineffective. The minimum number of tokens to transform is a hyper-parameter for the defense mechanism and is chosen such that at least 50% of the patch is covered by the transformed tokens. STRAP-ViT fits as a non-trainable plug-and-play block within the ViT architectures, for inference purposes only, with a minimal computational cost and does not require any additional training cost/effort. STRAP-ViT has been tested on multiple pre-trained vision transformer architectures (ViT-base-16 and DinoV2) and datasets (ImageNet and CalTech-101), across multiple adversarial attacks (Adversarial Patch, LAVAN, GDPA and RP2), and found to provide excellent robust accuracies lying within a 2-3% range of the clean baselines, and outperform the state-of-the-art.
Paper Structure (19 sections, 10 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 19 sections, 10 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Contributions
  4. Background and Related Work
  5. Vision Transformers
  6. Adversarial Patches
  7. Threat Model
  8. Literature Review: Adversarial Defenses
  9. STRAP-ViT Defense Mechanism
  10. Detection: Token Segregation
  11. Choosing $K$ Tokens:
  12. Mitigation: Token Transformation
  13. Results
  14. Key Findings
  15. Discussions
  16. ...and 4 more sections

Figures (4)

  • Figure 1: STRAP-ViT in action: Demonstrating the detection and transformation of tokens which overlap with the adversarial patch using the proposed STRAP-ViT defense technique. In this case, the minimum number of tokens to modify is two ($K=2$), such that at least $50\%$ of the patch is covered, to render the patch ineffective.
  • Figure 2: Illustration of the statistical separability of the tokens which overlap with the adversarial patch and other tokens from the clean part of the image, using entropy as a metric.
  • Figure 3: STRAP-ViT pipeline: The two-stage adversarial defense framework comprising of the Detection using Token Segregation through Jensen-Shannon Divergence and Mitigation using Token Transformation through composite randomised transformations, preceded by ViT Pre-processing Tokenization and succeeded by Vit Forward Pass.
  • Figure 4: A plot of the performance analysis of STRAP-ViT, with multiple adversarial patches ( Adversarial Patch, LAVAN, GDPA) and patch sizes, across two datasets (ImageNet and CalTech-101), also including the clean baseline and the impact of applying STRAP-ViT on clean samples, for reference and comparison.