Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models

Jiaxiang Liu, Jiawei Du, Xiao Liu, Prayag Tiwari, Mingkun Xu

TL;DR

This work tackles the vulnerability of vision-language models like CLIP to adversarial perturbations in zero-shot settings. It introduces Self-Calibrated Consistency (SCC), a plug-and-play test-time defense that unifies semantic consistency (soft pseudo-labels from multi-view counterattacks to repel hard negatives) and spatial consistency (multi-view aggregation with a shared corrective perturbation) to stabilize cross-modal alignment under attack. Through theoretical insight into semantic drift, view sensitivity, and hard-negative dominance, SCC demonstrates consistent robustness gains across 22 benchmarks and extends effectively to BioMedCLIP. The approach achieves strong robustness with minimal impact on clean accuracy and negligible runtime overhead, offering a practical path toward reliable vision-language systems in open-world and safety-critical applications.

Abstract

Pre-trained vision-language models (VLMs) such as CLIP have demonstrated strong zero-shot capabilities across diverse domains, yet remain highly vulnerable to adversarial perturbations that disrupt image-text alignment and compromise reliability. Existing defenses typically rely on adversarial fine-tuning with labeled data, limiting their applicability in zero-shot settings. In this work, we identify two key weaknesses of current CLIP adversarial attacks -- lack of semantic guidance and vulnerability to view variations -- collectively termed semantic and viewpoint fragility. To address these challenges, we propose Self-Calibrated Consistency (SCC), an effective test-time defense. SCC consists of two complementary modules: Semantic consistency, which leverages soft pseudo-labels from counterattack warm-up and multi-view predictions to regularize cross-modal alignment and separate the target embedding from confusable negatives; and Spatial consistency, aligning perturbed visual predictions via augmented views to stabilize inference under adversarial perturbations. Together, these modules form a plug-and-play inference strategy. Extensive experiments on 22 benchmarks under diverse attack settings show that SCC consistently improves the zero-shot robustness of CLIP while maintaining accuracy, and can be seamlessly integrated with other VLMs for further gains. These findings highlight the great potential of establishing an adversarially robust paradigm from CLIP, with implications extending to broader vision-language domains such as BioMedCLIP.

Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models

TL;DR

This work tackles the vulnerability of vision-language models like CLIP to adversarial perturbations in zero-shot settings. It introduces Self-Calibrated Consistency (SCC), a plug-and-play test-time defense that unifies semantic consistency (soft pseudo-labels from multi-view counterattacks to repel hard negatives) and spatial consistency (multi-view aggregation with a shared corrective perturbation) to stabilize cross-modal alignment under attack. Through theoretical insight into semantic drift, view sensitivity, and hard-negative dominance, SCC demonstrates consistent robustness gains across 22 benchmarks and extends effectively to BioMedCLIP. The approach achieves strong robustness with minimal impact on clean accuracy and negligible runtime overhead, offering a practical path toward reliable vision-language systems in open-world and safety-critical applications.

Abstract

Pre-trained vision-language models (VLMs) such as CLIP have demonstrated strong zero-shot capabilities across diverse domains, yet remain highly vulnerable to adversarial perturbations that disrupt image-text alignment and compromise reliability. Existing defenses typically rely on adversarial fine-tuning with labeled data, limiting their applicability in zero-shot settings. In this work, we identify two key weaknesses of current CLIP adversarial attacks -- lack of semantic guidance and vulnerability to view variations -- collectively termed semantic and viewpoint fragility. To address these challenges, we propose Self-Calibrated Consistency (SCC), an effective test-time defense. SCC consists of two complementary modules: Semantic consistency, which leverages soft pseudo-labels from counterattack warm-up and multi-view predictions to regularize cross-modal alignment and separate the target embedding from confusable negatives; and Spatial consistency, aligning perturbed visual predictions via augmented views to stabilize inference under adversarial perturbations. Together, these modules form a plug-and-play inference strategy. Extensive experiments on 22 benchmarks under diverse attack settings show that SCC consistently improves the zero-shot robustness of CLIP while maintaining accuracy, and can be seamlessly integrated with other VLMs for further gains. These findings highlight the great potential of establishing an adversarially robust paradigm from CLIP, with implications extending to broader vision-language domains such as BioMedCLIP.
Paper Structure (29 sections, 3 theorems, 21 equations, 8 figures, 8 tables, 1 algorithm)

This paper contains 29 sections, 3 theorems, 21 equations, 8 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Related Work
  3. Methodology
  4. The Findings of Test-time Counterattack
  5. The Analysis of Semantic and Spatial Fragility
  6. Mitigating Semantic Fragility via Semantic Consistency
  7. Cross-modal consistency.
  8. Soft prototype construction.
  9. Iterative counter-attack.
  10. Mitigating Spatial Fragility via Spatial Consistency
  11. Multi-view self-consistency.
  12. Confidence weighting.
  13. Spatial counterattacks.
  14. Objective.
  15. Experiments
  16. ...and 14 more sections

Key Result

Proposition 1

Let $x+\delta$ denote the counter-perturbed input during optimization. Optimizing $\mathcal{L}_{cm}$ by PGD ascent increases the semantic margin monotonically (up to $\mathcal{O}(\alpha^2)$), thereby preventing drift toward confusable negatives. See proof in Appendix.

Figures (8)

  • Figure 1: Analysis of Counterattack for Adversarial Robustness. Performance drops when reducing from two views to a single view, and degrades further under semantic perturbations.
  • Figure 2: During counterattack inference, embeddings tend to drift within the adversarial space and fall into hard-negative traps; SCC leverages cross-modal semantic and spatial consistency to push them away from hard samples and back toward the correct class space.
  • Figure 3: Test-time defense paradigms on CLIP. (a). R-TPT adapts text prompts online but still suffers from adversarial perturbations. (b). TTC repairs adversarial inputs via corrective perturbations, yet remains sensitive to view variance and hard negatives. (c). SCC enforces semantic and spatial consistency, yielding more stable recovery.
  • Figure 4: Pipeline of SCC: text and augmented views are encoded into features, multi-view embeddings are aggregated with averaging and combined with a short counterattack warm-up to yield stable soft pseudo-labels, which then guide cross-modal consistency optimization through the corrective perturbation $\delta$. $T_r$ denotes the correct class embedding (e.g., dog), while $T_f$ is an incorrect class embedding (e.g., wolf). Spatial consistency enforces perturbed views $f_\theta(x_i+\delta)$ to stay close to $T_r$ rather than drift toward $T_f$.
  • Figure 5: Sensitivity of SCC to $\lambda_{cm}$, number of views $L$, and effect of the temperature $t$ in soft-label sharpening (The $t$-axis in the plot is scaled by $\times 10$). A moderate $t$ yields the best trade-off.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Proposition 1: Hard-negative repulsion
  • Proposition 2: Variance reduction
  • Remark 1
  • Remark 2: Optimization coupling
  • Proposition 3: Suppression of spurious negatives
  • Remark 3