Enhancing CLIP Robustness via Cross-Modality Alignment

TL;DR

COLA addresses CLIP-like robustness gaps under adversarial perturbations by explicit cross-modality alignment. It first projects adversarial image embeddings onto a text-induced subspace $\mathcal{U}$ using $\Pi(\hat{\mathbf{x}})$ to restore global alignment, then refines cross-modal matching via optimal transport over multiple augmented views with a projected cost $\mathbf{C}^{\Pi}$ to enforce local semantic consistency. Theoretical analyses show cosine distortion is reduced by projection and OT margins are amplified, leading to better generalization. Empirically, COLA delivers consistent robustness gains across 14 zero-shot benchmarks with minimal impact on clean accuracy and favorable efficiency compared to training-time defenses, and it remains compatible with existing CLIP fine-tuned models. This makes COLA a practical, training-free defense for robust multimodal inference in high-stakes applications such as safety-critical vision-language systems.

Abstract

Vision-language models (VLMs) such as CLIP demonstrate strong generalization in zero-shot classification but remain highly vulnerable to adversarial perturbations. Existing methods primarily focus on adversarial fine-tuning or prompt optimization; they often overlook the gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other. This misalignment is significantly amplified under adversarial perturbations, leading to severe degradation in classification performance. To address this problem, we propose Cross-modality Alignment, dubbed COLA, an optimal transport-based framework that explicitly addresses adversarial misalignment by restoring both global image-text alignment and local structural consistency in the feature space. (1) COLA first projects adversarial image embeddings onto a subspace spanned by class text features, effectively filtering out non-semantic distortions while preserving discriminative information. (2) It then models images and texts as discrete distributions over multiple augmented views and refines their alignment via OT, with the subspace projection seamlessly integrated into the cost computation. This design ensures stable cross-modal alignment even under adversarial conditions. COLA is training-free and compatible with existing fine-tuned models. Extensive evaluations across 14 zero-shot classification benchmarks demonstrate the effectiveness of COLA, especially with an average improvement of 6.7% on ImageNet and its variants under PGD adversarial attacks, while maintaining high accuracy on clean samples.

Figures (4)

• Figure 1: Visualization of image and text feature distributions under different conditions. We plot text and image embeddings via Principal component analysis (PCA) abdi2010principal and compare their performance. (a) Adversarial perturbations cause image features to scatter and misalign with text features. (b) Clean image and text features naturally form two distinct clusters as a result of contrastive training. (c) Our method mitigates the misalignment, making adversarial image features closer to the text features. (d) Classification performance across multiple VLMs shows that our method consistently improves robustness against adversarial inputs.
• Figure 2: Accuracy (%) comparisons across Caltech101 and ImageNet datasets with varying the number of augmentations. (a) and (b): Classification results under different numbers of image augmentations. (c) and (d): Classification results under different numbers of class name augmentations.
• Figure 3: Accuracy (%) with different numbers of principal components in the projection matrix.
• Figure 4: Similarity distributions among original, attacked, and projected features on ImageNet.

Theorems & Definitions (1)

• proof