Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Filtered-ViT: A Robust Defense Against Multiple Adversarial Patch Attacks

Aja Khanal, Ahmed Faid, Apurva Narayan

TL;DR

This work tackles the vulnerability of vision systems to localized adversarial patches, especially under multi-patch attacks, by introducing Filtered-ViT, which embeds SMART-VMF to selectively suppress corrupted regions before derandomized smoothing within a Vision Transformer backbone. The method combines adaptive weighting, multi-scale analysis, and reliability-driven fusion to achieve strong robustness without sacrificing clean accuracy. On ImageNet with LaVAN patches, Filtered-ViT attains 79.8% clean and 46.3% robust accuracy under four simultaneous 1% patches, outperforming prior defenses, and extends to medical imaging where it mitigates scanner noise and occlusions while preserving diagnostic content. The approach offers a principled, hardware-friendly path toward trustworthy vision systems in high-stakes settings by unifying adversarial and natural patch robustness within a single transformer-based framework.

Abstract

Deep learning vision systems are increasingly deployed in safety-critical domains such as healthcare, yet they remain vulnerable to small adversarial patches that can trigger misclassifications. Most existing defenses assume a single patch and fail when multiple localized disruptions occur, the type of scenario adversaries and real-world artifacts often exploit. We propose Filtered-ViT, a new vision transformer architecture that integrates SMART Vector Median Filtering (SMART-VMF), a spatially adaptive, multi-scale, robustness-aware mechanism that enables selective suppression of corrupted regions while preserving semantic detail. On ImageNet with LaVAN multi-patch attacks, Filtered-ViT achieves 79.8% clean accuracy and 46.3% robust accuracy under four simultaneous 1\% patches, outperforming existing defenses. Beyond synthetic benchmarks, a real-world case study on radiographic medical imagery shows that Filtered-ViT mitigates natural artifacts such as occlusions and scanner noise without degrading diagnostic content. This establishes Filtered-ViT as the first transformer to demonstrate unified robustness against both adversarial and naturally occurring patch-like disruptions, charting a path toward reliable vision systems in truly high-stakes environments.

Filtered-ViT: A Robust Defense Against Multiple Adversarial Patch Attacks

TL;DR

This work tackles the vulnerability of vision systems to localized adversarial patches, especially under multi-patch attacks, by introducing Filtered-ViT, which embeds SMART-VMF to selectively suppress corrupted regions before derandomized smoothing within a Vision Transformer backbone. The method combines adaptive weighting, multi-scale analysis, and reliability-driven fusion to achieve strong robustness without sacrificing clean accuracy. On ImageNet with LaVAN patches, Filtered-ViT attains 79.8% clean and 46.3% robust accuracy under four simultaneous 1% patches, outperforming prior defenses, and extends to medical imaging where it mitigates scanner noise and occlusions while preserving diagnostic content. The approach offers a principled, hardware-friendly path toward trustworthy vision systems in high-stakes settings by unifying adversarial and natural patch robustness within a single transformer-based framework.

Abstract

Deep learning vision systems are increasingly deployed in safety-critical domains such as healthcare, yet they remain vulnerable to small adversarial patches that can trigger misclassifications. Most existing defenses assume a single patch and fail when multiple localized disruptions occur, the type of scenario adversaries and real-world artifacts often exploit. We propose Filtered-ViT, a new vision transformer architecture that integrates SMART Vector Median Filtering (SMART-VMF), a spatially adaptive, multi-scale, robustness-aware mechanism that enables selective suppression of corrupted regions while preserving semantic detail. On ImageNet with LaVAN multi-patch attacks, Filtered-ViT achieves 79.8% clean accuracy and 46.3% robust accuracy under four simultaneous 1\% patches, outperforming existing defenses. Beyond synthetic benchmarks, a real-world case study on radiographic medical imagery shows that Filtered-ViT mitigates natural artifacts such as occlusions and scanner noise without degrading diagnostic content. This establishes Filtered-ViT as the first transformer to demonstrate unified robustness against both adversarial and naturally occurring patch-like disruptions, charting a path toward reliable vision systems in truly high-stakes environments.

Paper Structure

This paper contains 39 sections, 14 equations, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Adversarial Patch
  4. LaVAN: Localized and Visible Adversarial Noise
  5. Vision Transformers (ViTs)
  6. (De)randomized Smoothing
  7. Smoothed-ViT
  8. Vector Median Filter (VMF)
  9. Adversarial Attacks in Medical Imaging
  10. Methodology
  11. Patch Training
  12. Patch Placement
  13. Filtered-ViT Architecture
  14. SMART-VMF.
  15. Obtaining Robust and Clean Accuracies
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Example of adversarial patches placed at image corners.
  • Figure 2: Filtered-ViT Architecture: SMART-VMF first filters adversarial or artifact-heavy regions, followed by derandomized smoothing and classification using a Vision Transformer.
  • Figure 3: Accuracy trends across patch size and quantity for all models. Top: Robust and Clean accuracy vs. patch size. Bottom: Robust and Clean accuracy vs. patch quantity.
  • Figure 4: Chest radiograph with cardiac pacemaker
  • Figure 5: Axial GRE MRI with susceptibility artifact mimicking a cerebral microbleed.