Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detection

Yinghe Zhang, Chi Liu, Shuai Zhou, Sheng Shen, Peng Gui

TL;DR

This work tackles the generalization gap in adversarial attack detection by reframing detection as anomaly detection over natural image distributions and leveraging a parameter-efficient, multimodal CLIP framework. By jointly fine-tuning a visual adapter and learnable textual prompts within a CLIP-Fusion architecture, the method achieves robust detection across multiple gradient-based attacks with substantially fewer trainable parameters than traditional CNN-based detectors. Extensive experiments demonstrate strong cross-attack generalization, high detectability, and faster, more stable training, while maintaining robustness to varying perturbation magnitudes. The approach offers a practical, scalable defense for real-world vision systems against evolving adversarial threats.

Abstract

Adversarial attacks pose a critical security threat to real-world AI systems by injecting human-imperceptible perturbations into benign samples to induce misclassification in deep learning models. While existing detection methods, such as Bayesian uncertainty estimation and activation pattern analysis, have achieved progress through feature engineering, their reliance on handcrafted feature design and prior knowledge of attack patterns limits generalization capabilities and incurs high engineering costs. To address these limitations, this paper proposes a lightweight adversarial detection framework based on the large-scale pre-trained vision-language model CLIP. Departing from conventional adversarial feature characterization paradigms, we innovatively adopt an anomaly detection perspective. By jointly fine-tuning CLIP's dual visual-text encoders with trainable adapter networks and learnable prompts, we construct a compact representation space tailored for natural images. Notably, our detection architecture achieves substantial improvements in generalization capability across both known and unknown attack patterns compared to traditional methods, while significantly reducing training overhead. This study provides a novel technical pathway for establishing a parameter-efficient and attack-agnostic defense paradigm, markedly enhancing the robustness of vision systems against evolving adversarial threats.

Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detection

TL;DR

This work tackles the generalization gap in adversarial attack detection by reframing detection as anomaly detection over natural image distributions and leveraging a parameter-efficient, multimodal CLIP framework. By jointly fine-tuning a visual adapter and learnable textual prompts within a CLIP-Fusion architecture, the method achieves robust detection across multiple gradient-based attacks with substantially fewer trainable parameters than traditional CNN-based detectors. Extensive experiments demonstrate strong cross-attack generalization, high detectability, and faster, more stable training, while maintaining robustness to varying perturbation magnitudes. The approach offers a practical, scalable defense for real-world vision systems against evolving adversarial threats.

Abstract

Adversarial attacks pose a critical security threat to real-world AI systems by injecting human-imperceptible perturbations into benign samples to induce misclassification in deep learning models. While existing detection methods, such as Bayesian uncertainty estimation and activation pattern analysis, have achieved progress through feature engineering, their reliance on handcrafted feature design and prior knowledge of attack patterns limits generalization capabilities and incurs high engineering costs. To address these limitations, this paper proposes a lightweight adversarial detection framework based on the large-scale pre-trained vision-language model CLIP. Departing from conventional adversarial feature characterization paradigms, we innovatively adopt an anomaly detection perspective. By jointly fine-tuning CLIP's dual visual-text encoders with trainable adapter networks and learnable prompts, we construct a compact representation space tailored for natural images. Notably, our detection architecture achieves substantial improvements in generalization capability across both known and unknown attack patterns compared to traditional methods, while significantly reducing training overhead. This study provides a novel technical pathway for establishing a parameter-efficient and attack-agnostic defense paradigm, markedly enhancing the robustness of vision systems against evolving adversarial threats.

Paper Structure

This paper contains 20 sections, 5 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. RELATED WORKS
  3. Adversarial Attacks and Adversarial Example Detection
  4. Pre-trained Models and Fine-Tuning Methods
  5. METHODOLOGY
  6. CLIP’s Generalization Advantage and Transfer Learning
  7. 3.2.1 Visual Adapter
  8. 3.2.2 Prompt Tuning
  9. 3.2.3 Fusion Architecture
  10. Experiment
  11. General Setup
  12. Attack Methods and Datasets
  13. Baselines
  14. Detectability
  15. Generalization Performance
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Static decision boundary isolation vs dynamic distribution generalization
  • Figure 2: Flowcharts of different fine-tuning methods for pretrained Clip encoders: i) Visual Adapter; ii) Prompt Tuning; iii) our fusion architecture that incorporates Visual Adapter and Prompt Tuning for lightweight and generalized adversarial example detection, as shown in the black dash box. Visual adaptor or prompt tuning can be viewed as a simplified version of our fusion model, where only one modality is fine-tuned while the other remains frozen.
  • Figure 3: Visual comparison between natural images and eight adversarial examples. The perturbations are human-imperceptible.
  • Figure 4: Comparison of training loss curves between a ResNet18 detector and ours.