Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP

TL;DR

Zero-shot CLIP is highly vulnerable to adversarial examples. The authors reveal a spectral fragility: adversarial signals rely on mid-to-high frequencies due to CLIP's spectral bias, while benign content remains stable under frequency attenuation. They propose CSR, a test-time defense comprising spectral-consistency-based detection and a contrastive rectification that guides inputs toward the natural manifold without retraining. Across 16 datasets, CSR achieves state-of-the-art robustness under strong attacks (e.g., +18.1% under AutoAttack) with modest latency and generalizes to segmentation, captioning, and VQA, illustrating its broad practicality. This work offers a practical, universal defense blueprint for CLIP-based systems and LVLM ecosystems.

Abstract

Vision-language models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, yet remain highly vulnerable to adversarial examples (AEs). While test-time defenses are promising, existing methods fail to provide sufficient robustness against strong attacks and are often hampered by high inference latency and task-specific applicability. To address these limitations, we start by investigating the intrinsic properties of AEs, which reveals that AEs exhibit severe feature inconsistency under progressive frequency attenuation. We further attribute this to the model's inherent spectral bias. Leveraging this insight, we propose an efficient test-time defense named Contrastive Spectral Rectification (CSR). CSR optimizes a rectification perturbation to realign the input with the natural manifold under a spectral-guided contrastive objective, which is applied input-adaptively. Extensive experiments across 16 classification benchmarks demonstrate that CSR outperforms the SOTA by an average of 18.1% against strong AutoAttack with modest inference overhead. Furthermore, CSR exhibits broad applicability across diverse visual tasks. Code is available at https://github.com/Summu77/CSR.

Figures (10)

• Figure 1: Zero-shot adversarial robustness comparison. We evaluate our CSR against CLIP, FARE (adversarial fine-tuning), and TTC (test-time defense) on 16 datasets grouped into General, Fine-Grained (FG), Scene, and Domain. The radar charts show Top-1 accuracy on benign and adversarial samples under standard PGD ($\ell_\infty=1/255$) and the stronger AutoAttack ($\ell_\infty=4/255$).
• Figure 2: Mechanism of Contrastive Spectral Rectification (CSR). Leveraging a spectral contrastive strategy, CSR exerts repulsion from the original adversarial feature (solid red star) within the adversarial subspace, while inducing attraction toward the low-pass filtered feature (dashed red star)—an approximation on the benign manifold. This synergy steers the optimization toward the ground-truth feature (solid blue star), effectively rectifying the feature space as corroborated by the PCA visualization (right) on 300 images.
• Figure 3: Spectral Consistency Disparity. Cosine similarity between original and low-pass filtered embeddings across decaying bandwidth radii $r$. While benign examples maintain high semantic fidelity, adversarial examples exhibit rapid feature collapse. See Appendix \ref{['APP: SCD']} for consistent trends across additional datasets.
• Figure 4: Analysis. (a) The gradient magnitude (left) and representational shift (right) are concentrated in mid-to-high frequency components, indicating that the model is most vulnerable to perturbations in these bands. (b) Attacks constrained to low frequencies are inefficient. (c) The adversarial gradient exhibits consistent negative cosine similarity with the low-frequency constraint gradient.
• Figure 5: Ablation study on the rectification steps $N$, the Gaussian filter radius $r$, and the detection threshold $\tau$. More in Appendix \ref{['app:ablstep']}