FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models

TL;DR

This work addresses CLIP's vulnerability to imperceptible adversarial perturbations by introducing FPT-Noise, a training-free test-time defense. It combines a Dynamic Feature Modulator, a Feature Perception Threshold for attack discrimination, and a Scene-Aware Regulation with Test-Time Transformation Ensembling to adaptively inject counter-noise and mitigate residual artifacts. Empirical results across 16 datasets and multiple CLIP backbones show a dramatic boost in robust accuracy under AutoAttack (0.07% to 56.86%), while preserving high zero-shot performance on clean data. The approach offers a practical, efficient defense that preserves zero-shot generalization and obviates costly fine-tuning, with potential applicability to broader vision-language systems.

Abstract

Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalizability across diverse downstream tasks. However, recent studies have revealed that VLMs, including CLIP, are highly vulnerable to adversarial attacks, particularly on their visual modality. Traditional methods for improving adversarial robustness, such as adversarial training, involve extensive retraining and can be computationally expensive. In this paper, we propose a new Test-Time defense: Feature Perception Threshold Counterattack Noise (FPT-Noise), which enhances the adversarial robustness of CLIP without costly fine-tuning. Our core contributions are threefold: First, we introduce a Dynamic Feature Modulator that dynamically generate an image-specific and attack-adaptive noise intensity parameter. Second, We reanalyzed the image features of CLIP. When images are exposed to different levels of noise, clean images and adversarial images exhibit distinct rates of feature change. We established a feature perception threshold to distinguish clean images from attacked ones. Finally, we integrate a Scene-Aware Regulation guided by a stability threshold and leverage Test-Time Transformation Ensembling (TTE) to further mitigate the impact of residual noise and enhance robustness.Extensive experimentation has demonstrated that FPT-Noise significantly outperforms existing Test-Time defense methods, boosting average robust accuracy from 0.07% to 56.86% under AutoAttack while maintaining high performance on clean images (-1.1%). The code will be made public following the publication of the study. The code will be made public following the publication of the study.

Figures (4)

• Figure 1: (a) We tested the defensive efficacy of TTC and FPT-Noise against PGD-10 with different perturbation budgets averaged on 16 datasets. (b) Classification accuracy (%) on both adversarial images (Robs.) under PGD-10 attack at $\epsilon=1/255$.
• Figure 2: The pipeline of FPT-Noise. Pre-trained CLIP acquires text embeddings and image features $f_\theta(X)$, which are processed by the Dynamic Feature Modulator ($M$). The counterattack noise, denoted by $\delta_{c}$, is generated via the loss function. Using our designed Feature Perception Threshold and Scene-Aware Regulation, control the intensity of the noise, then add it to the image and then TTE is performed. The Softmax function, when integrated with text embeddings, yields the desired output.
• Figure 3: Average $\tau$ across 16 image datasets.
• Figure 4: We utilize CLIP to compute the feature norm $||f_\theta(X+\delta_c)||$ of images with added FPT-Noise and the feature norm $||f_\theta(X)||$ of initial images. We selected three datasets from different domains for analysis.