Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models

Xiaojun Jia, Sensen Gao, Simeng Qin, Ke Ma, Xinfeng Li, Yihao Huang, Wei Dong, Yang Liu, Xiaochun Cao

TL;DR

This work tackles the vulnerability of large vision-language models to adversarial examples by introducing ER-APT, an evolution-based region adversarial prompt learning framework. It merges gradient-based adversarial generation with a genetic-evolution refinement to produce diverse, region-focused perturbations for few-shot prompt tuning, complemented by a dynamic loss weighting scheme that adaptively balances clean accuracy and robustness. The approach achieves state-of-the-art robustness and generalization across multiple datasets and settings, including cross-dataset and AutoAttack evaluations, with ablations confirming the critical roles of the evolutionary region and adaptive loss weighting. Although it incurs additional training cost, ER-APT provides a practical and scalable route to robust VLM deployment and transferability across tasks and data distributions.

Abstract

Large pre-trained vision-language models (VLMs), such as CLIP, demonstrate impressive generalization but remain highly vulnerable to adversarial examples (AEs). Previous work has explored robust text prompts through adversarial training, achieving some improvement in both robustness and generalization. However, they primarily rely on singlegradient direction perturbations (e.g., PGD) to generate AEs, which lack diversity, resulting in limited improvement in adversarial robustness. To address these limitations, we propose an evolution-based region adversarial prompt tuning method called ER-APT, which combines gradient methods with genetic evolution to generate more diverse and challenging AEs. In each training iteration, we first generate AEs using traditional gradient-based methods. Subsequently, a genetic evolution mechanism incorporating selection, mutation, and crossover is applied to optimize the AEs, ensuring a broader and more aggressive perturbation distribution.The final evolved AEs are used for prompt tuning, achieving region-based adversarial optimization instead of conventional single-point adversarial prompt tuning. We also propose a dynamic loss weighting method to adjust prompt learning efficiency for accuracy and robustness. Experimental evaluations on various benchmark datasets demonstrate the superiority of our proposed method, outperforming stateof-the-art APT methods. The code is released at https://github.com/jiaxiaojunQAQ/ER-APT.

Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models

TL;DR

This work tackles the vulnerability of large vision-language models to adversarial examples by introducing ER-APT, an evolution-based region adversarial prompt learning framework. It merges gradient-based adversarial generation with a genetic-evolution refinement to produce diverse, region-focused perturbations for few-shot prompt tuning, complemented by a dynamic loss weighting scheme that adaptively balances clean accuracy and robustness. The approach achieves state-of-the-art robustness and generalization across multiple datasets and settings, including cross-dataset and AutoAttack evaluations, with ablations confirming the critical roles of the evolutionary region and adaptive loss weighting. Although it incurs additional training cost, ER-APT provides a practical and scalable route to robust VLM deployment and transferability across tasks and data distributions.

Abstract

Large pre-trained vision-language models (VLMs), such as CLIP, demonstrate impressive generalization but remain highly vulnerable to adversarial examples (AEs). Previous work has explored robust text prompts through adversarial training, achieving some improvement in both robustness and generalization. However, they primarily rely on singlegradient direction perturbations (e.g., PGD) to generate AEs, which lack diversity, resulting in limited improvement in adversarial robustness. To address these limitations, we propose an evolution-based region adversarial prompt tuning method called ER-APT, which combines gradient methods with genetic evolution to generate more diverse and challenging AEs. In each training iteration, we first generate AEs using traditional gradient-based methods. Subsequently, a genetic evolution mechanism incorporating selection, mutation, and crossover is applied to optimize the AEs, ensuring a broader and more aggressive perturbation distribution.The final evolved AEs are used for prompt tuning, achieving region-based adversarial optimization instead of conventional single-point adversarial prompt tuning. We also propose a dynamic loss weighting method to adjust prompt learning efficiency for accuracy and robustness. Experimental evaluations on various benchmark datasets demonstrate the superiority of our proposed method, outperforming stateof-the-art APT methods. The code is released at https://github.com/jiaxiaojunQAQ/ER-APT.

Paper Structure

This paper contains 26 sections, 1 theorem, 24 equations, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt tuning for VLMs
  4. Adversarial Prompt Tuning in VLMs
  5. The Proposed Method
  6. Preliminary Study
  7. Pipeline of The Proposed Method
  8. Formulation of The Proposed Method
  9. Proposed Dynamic Loss Weighting Method
  10. Theoretical Analysis
  11. Experiments
  12. Setups
  13. Adversarial few-shot learning
  14. Adversarial base-to-new generalization
  15. Adversarial Cross-Dataset Evaluation
  16. ...and 11 more sections

Key Result

Theorem 1

Suppose that the loss function $\mathcal{L}$ is $L$-Lipschitz in $\boldsymbol{P}$, i.e. where $\boldsymbol{\Delta}_{\epsilon}$ is the allowable perturbation space as If ER-APT eq:objective optimizes the learnable prompts $\boldsymbol{P}$ to satisfy then for any perturbation $\delta\in\boldsymbol{\Delta}_{\epsilon}$, we have where $\eta$ satisfies there exists $\delta_i\in\mathcal{P}$ such that

Figures (3)

  • Figure 1: FAP vs ER-APT. (a) illustrates the adversarial examples generation in FAP zhou2024fewshot, which adopts the single-gradient direction to generate adversarial examples. (b) illustrates the generation of adversarial examples in our ER-APT, which combines gradient-based adversarial example generation with genetic evolution. (c) compares robustness of our ER-APT against FAP.
  • Figure 2: The Pipeline of the Proposed ER-APT. (a) Pipeline for the evolution-based region in adversarial example generation. (b) Pipeline for the generation of the text and image features. (c) Pipeline for the dynamic loss weighting.
  • Figure 3: Ablation Study. The two innovations, Evolution-based Region and Dynamic Loss Weighting, are each ablated to compare their adversarial base-to-new generalization.

Theorems & Definitions (1)

  • Theorem 1