MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Models

TL;DR

Vision-Language Models remain vulnerable to adversarial inputs despite strong generalization. The paper introduces Mixture of Adversarial Prompt Tuning (MoAPT), which learns multiple short base prompts and a conditional weight router to produce sample-specific, robust text features without full-parameter fine-tuning. Empirical results across 11 datasets show MoAPT yields notable robustness gains under PGD and AutoAttack, with favorable accuracy-robustness trade-offs and strong cross-dataset transferability. The work advances robust prompt-tuning by demonstrating that a mixture-prompt strategy with adaptive weighting outperforms longer single prompts and prior methods, offering an efficient defense for VLMs.

Abstract

Large pre-trained Vision Language Models (VLMs) demonstrate excellent generalization capabilities but remain highly susceptible to adversarial examples, posing potential security risks. To improve the robustness of VLMs against adversarial examples, adversarial prompt tuning methods are proposed to align the text feature with the adversarial image feature without changing model parameters. However, when facing various adversarial attacks, a single learnable text prompt has insufficient generalization to align well with all adversarial image features, which ultimately results in overfitting. To address the above challenge, in this paper, we empirically find that increasing the number of learned prompts yields greater robustness improvements than simply extending the length of a single prompt. Building on this observation, we propose an adversarial tuning method named \textbf{Mixture of Adversarial Prompt Tuning (MoAPT)} to enhance the generalization against various adversarial attacks for VLMs. MoAPT aims to learn mixture text prompts to obtain more robust text features. To further enhance the adaptability, we propose a conditional weight router based on the adversarial images to predict the mixture weights of multiple learned prompts, which helps obtain sample-specific mixture text features aligning with different adversarial image features. Extensive experiments across 11 datasets under different settings show that our method can achieve better adversarial robustness than state-of-the-art approaches.

Figures (3)

• Figure 1: The framework of Mixture of Adversarial Prompt Tuning (MoAPT). To enhance the adversarial robustness, we apply adversarial mixture prompt to generate diverse individual text feature, and utilize the conditional prompt weight router to obtain a sample-specific mixture text feature, and finally bring more generalization towards different adversarial examples.
• Figure 2: The performance of adversarial prompt tuning with different length and number on five datasets. "APT-Lm-Nk" denotes the APT with prompt length m and prompt number k. We find that increasing the number of prompts can enhance more robustness than increasing the prompt length (i.e., solid lines show better performance than dotted lines).
• Figure 3: Trade-off between Accuracy and Robustness ($M=16$).

Theorems & Definitions (1)

• Theorem 1