Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference

TL;DR

This paper addresses the vulnerability of vision-language models like CLIP to adversarial perturbations and the limited diversity of existing test-time defenses. It introduces Directional Orthogonal Counterattack (DOC), which augments counterattack updates with orthogonal gradient exploration and momentum, and a directional sensitivity score to adapt counterattack strength. Empirical results across 16 datasets show that DOC significantly improves robustness under PGD, CW, and AutoAttack while maintaining competitive clean accuracy, and it can further enhance adversarially fine-tuned models. The approach is lightweight, training-free, and transferable to other multimodal systems, offering a scalable defense for real-world vision-language applications.

Abstract

Vision-language pre-training models (VLPs) demonstrate strong multimodal understanding and zero-shot generalization, yet remain vulnerable to adversarial examples, raising concerns about their reliability. Recent work, Test-Time Counterattack (TTC), improves robustness by generating perturbations that maximize the embedding deviation of adversarial inputs using PGD, pushing them away from their adversarial representations. However, due to the fundamental difference in optimization objectives between adversarial attacks and counterattacks, generating counterattacks solely based on gradients with respect to the adversarial input confines the search to a narrow space. As a result, the counterattacks could overfit limited adversarial patterns and lack the diversity to fully neutralize a broad range of perturbations. In this work, we argue that enhancing the diversity and coverage of counterattacks is crucial to improving adversarial robustness in test-time defense. Accordingly, we propose Directional Orthogonal Counterattack (DOC), which augments counterattack optimization by incorporating orthogonal gradient directions and momentum-based updates. This design expands the exploration of the counterattack space and increases the diversity of perturbations, which facilitates the discovery of more generalizable counterattacks and ultimately improves the ability to neutralize adversarial perturbations. Meanwhile, we present a directional sensitivity score based on averaged cosine similarity to boost DOC by improving example discrimination and adaptively modulating the counterattack strength. Extensive experiments on 16 datasets demonstrate that DOC improves adversarial robustness under various attacks while maintaining competitive clean accuracy. Code is available at https://github.com/bookman233/DOC.

Figures (5)

• Figure 1: (a)-(b) Conceptual illustration of our methodology. We propose to generate more diverse counterattacks to neutralize adversarial perturbations. (c)-(d) t-SNE of example embeddings obtained by TTC and our DOC.
• Figure 2: (a)-(c) t-SNE visualizations of counterattacks generated by TTC and our DOC. (Bottom) Comparison of mean cosine similarity of counterattack and robust accuracy under PGD-10 with $\epsilon_{\text{atk}} = 4/255$. More details on 15 datasets are presented in Supplementary Materials.
• Figure 3: Performance of DOC combined with adversarial fine-tuning, including TeCoA maounderstanding, PMG-AFT wang2024pre, and FARE schlarmann2024robust. Robust accuracy is evaluated on 16 datasets using PGD-10 with $\epsilon_{\text{atk}} = 1/255$.
• Figure 4: Performance of DOC and other baselines under AutoAttack with a perturbation budget of $\epsilon_{\text{atk}} = 4/255$. Clean and robust accuracy is averaged across 16 datasets.
• Figure 5: Performance with numbers of counterattack steps $N$ on different datasets. Robust accuracy is evaluated by PGD-10 and CW with the perturbation budget $\epsilon_{\text{atk}} = 4/255$. Results on remaining datasets are presented in Supplementary Materials.