Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Defense in Vision-Language Models: An Overview

Xiaowei Fu, Lei Zhang

TL;DR

Vision-Language Models like CLIP are vulnerable to imperceptible adversarial perturbations within a budget $\epsilon$ (e.g., $1/255$), motivating a structured defense study. The paper surveys a taxonomy of defenses—Training-time Defense, Test-time Adaptation, and Training-free Defense—analyzing methods, challenges, and empirical trends across multiple datasets. It finds that training-free defenses can offer competitive robustness and efficiency (with COLA demonstrating strong performance), while training-time methods deliver robustness at higher computational cost. The work highlights cross-modal attack challenges and points to promising directions such as adaptive defenses, semi-supervised approaches, and integrating generative models to enhance resilience in real-world deployment.

Abstract

The widespread use of Vision Language Models (VLMs, e.g. CLIP) has raised concerns about their vulnerability to sophisticated and imperceptible adversarial attacks. These attacks could compromise model performance and system security in cross-modal tasks. To address this challenge, three main defense paradigms have been proposed: Training-time Defense, Test-time Adaptation Defense, and Training-free Defense. Training-time Defense involves modifying the training process, typically through adversarial fine-tuning to improve the robustness to adversarial examples. While effective, this approach requires substantial computational resources and may not generalize across all adversarial attacks. Test-time Adaptation Defense focuses on adapting the model at inference time by updating its parameters to handle unlabeled adversarial examples, offering flexibility but often at the cost of increased complexity and computational overhead. Training-free Defense avoids modifying the model itself, instead focusing on altering the adversarial inputs or their feature embeddings, which enforces input perturbations to mitigate the impact of attacks without additional training. This survey reviews the latest advancements in adversarial defense strategies for VLMs, highlighting the strengths and limitations of such approaches and discussing ongoing challenges in enhancing the robustness of VLMs.

Adversarial Defense in Vision-Language Models: An Overview

TL;DR

Vision-Language Models like CLIP are vulnerable to imperceptible adversarial perturbations within a budget (e.g., ), motivating a structured defense study. The paper surveys a taxonomy of defenses—Training-time Defense, Test-time Adaptation, and Training-free Defense—analyzing methods, challenges, and empirical trends across multiple datasets. It finds that training-free defenses can offer competitive robustness and efficiency (with COLA demonstrating strong performance), while training-time methods deliver robustness at higher computational cost. The work highlights cross-modal attack challenges and points to promising directions such as adaptive defenses, semi-supervised approaches, and integrating generative models to enhance resilience in real-world deployment.

Abstract

The widespread use of Vision Language Models (VLMs, e.g. CLIP) has raised concerns about their vulnerability to sophisticated and imperceptible adversarial attacks. These attacks could compromise model performance and system security in cross-modal tasks. To address this challenge, three main defense paradigms have been proposed: Training-time Defense, Test-time Adaptation Defense, and Training-free Defense. Training-time Defense involves modifying the training process, typically through adversarial fine-tuning to improve the robustness to adversarial examples. While effective, this approach requires substantial computational resources and may not generalize across all adversarial attacks. Test-time Adaptation Defense focuses on adapting the model at inference time by updating its parameters to handle unlabeled adversarial examples, offering flexibility but often at the cost of increased complexity and computational overhead. Training-free Defense avoids modifying the model itself, instead focusing on altering the adversarial inputs or their feature embeddings, which enforces input perturbations to mitigate the impact of attacks without additional training. This survey reviews the latest advancements in adversarial defense strategies for VLMs, highlighting the strengths and limitations of such approaches and discussing ongoing challenges in enhancing the robustness of VLMs.
Paper Structure (10 sections, 2 equations, 1 figure, 2 tables)

This paper contains 10 sections, 2 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Image Prediction and Attack Paradigm of CLIP
  3. The Taxonomy of Defense Paradigms
  4. Training-time defense paradigm
  5. Adversarial Fine-tuning
  6. Adversarial Prompt Tuning
  7. Test-time adaptation defense paradigm
  8. Training-free defense paradigm
  9. Experiments
  10. Conclusion and Future Work

Figures (1)

  • Figure 1: The basic processes of three different defense paradigms. Training-time defense and Test-time Adaptation defense update model parameters during the training and testing phases, respectively, while Training-free defense keeps the parameters unchanged.