NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models

TL;DR

This work highlights the efficacy of internal feature-level intervention in prompt tuning for adversarial robustness, moving beyond input-side alignment approaches to create an adaptive defense mechanism that can identify and rectify adversarial perturbations across embedding spaces.

Abstract

Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable capabilities in understanding relationships between visual and textual data through joint embedding spaces. Despite their effectiveness, these models remain vulnerable to adversarial attacks, particularly in the image modality, posing significant security concerns. Building upon our previous work on Adversarial Prompt Tuning (AdvPT), which introduced learnable text prompts to enhance adversarial robustness in VLMs without extensive parameter training, we present a significant extension by introducing the Neural Augmentor framework for Multi-modal Adversarial Prompt Tuning (NAP-Tuning).Our key innovations include: (1) extending AdvPT from text-only to multi-modal prompting across both text and visual modalities, (2) expanding from single-layer to multi-layer prompt architectures, and (3) proposing a novel architecture-level redesign through our Neural Augmentor approach, which implements feature purification to directly address the distortions introduced by adversarial attacks in feature space. Our NAP-Tuning approach incorporates token refiners that learn to reconstruct purified features through residual connections, allowing for modality-specific and layer-specific feature correction.Comprehensive experiments demonstrate that NAP-Tuning significantly outperforms existing methods across various datasets and attack types. Notably, our approach shows significant improvements over the strongest baselines under the challenging AutoAttack benchmark, outperforming them by 33.5% on ViT-B16 and 33.0% on ViT-B32 architectures while maintaining competitive clean accuracy.

Figures (12)

• Figure 1: Comparison of adversarial prompt tuning approaches: (a) The original AdvPT that uses only text prompts, (b) Existing multimodal approaches that incorporate prompts in both visual and text pathways, and (c) Our proposed NAP-Tuning framework that extends multimodal prompting with feature purification via token refiners, enabling the reconstruction of clean feature representations from adversarially perturbed inputs.
• Figure 2: Overview of our proposed Neural Augmentor module for multi-modal AdvPT.
• Figure 3: Performance comparison between NAP-Tuning (frozen image encoder) and TeCoA (fine-tuned image encoder).
• Figure 4: Clean and robust accuracy across datasets when varying the number of prompt layers (1-12). Complex datasets (ImageNet, Food101, SUN397) show optimal performance at intermediate depth, while other datasets benefit from deeper architectures.
• Figure 5: Ablation analysis comparing models with and without the Neural Augmentor (NA) across varying prompt depths. The complete model consistently outperforms the ablated variant.