Semantic-aware Adversarial Fine-tuning for CLIP

TL;DR

Semantic-aware Adversarial Fine-Tuning (SAFT), which fine-tunes CLIP's image encoder with semantic-aware AEs, and outperforms current methods, achieving substantial improvements in zero-shot adversarial robustness across 16 datasets.

Abstract

Recent studies have shown that CLIP model's adversarial robustness in zero-shot classification tasks can be enhanced by adversarially fine-tuning its image encoder with adversarial examples (AEs), which are generated by minimizing the cosine similarity between images and a hand-crafted template (e.g., ''A photo of a {label}''). However, it has been shown that the cosine similarity between a single image and a single hand-crafted template is insufficient to measure the similarity for image-text pairs. Building on this, in this paper, we find that the AEs generated using cosine similarity may fail to fool CLIP when the similarity metric is replaced with semantically enriched alternatives, making the image encoder fine-tuned with these AEs less robust. To overcome this issue, we first propose a semantic-ensemble attack to generate semantic-aware AEs by minimizing the average similarity between the original image and an ensemble of refined textual descriptions. These descriptions are initially generated by a foundation model to capture core semantic features beyond hand-crafted templates and are then refined to reduce hallucinations. To this end, we propose Semantic-aware Adversarial Fine-Tuning (SAFT), which fine-tunes CLIP's image encoder with semantic-aware AEs. Extensive experiments show that SAFT outperforms current methods, achieving substantial improvements in zero-shot adversarial robustness across 16 datasets. Our code is available at: https://github.com/tmlr-group/SAFT.

Figures (3)

• Figure 1: Comparison between CLIP radford2021learning, CuPL pratt2023what and WCA li2024visual as similarity metrics for clean images and their corresponding AEs, generated by minimizing the CLIP score via projected gradient descent (PGD) madry2018towards, across six animal classes in ImageNet-1K deng2009imagenet. Points below the diagonal line indicate a success in attacking the similarity metric. The results show that although these AEs can reduce the CLIP score, they may fail to fool CLIP when more semantically enriched scores are used as alternatives. This observation motivates us to rethink how AEs should be constructed in the case of CLIP.
• Figure 2: An overview of SAFT. In hallucination-aware description generation, a foundation model generates diverse textual descriptions for each class label, followed by the semantic filtering strategy to retain top-K most relevant descriptions. These refined descriptions are then encoded by CLIP's text encoder. In semantic-ensemble attack, AEs are generated by maximizing misalignment between the visual embeddings and the average embeddings of refined descriptions. Finally, the image encoder is fine-tuned by minimizing this misalignment, aiming to learn linguistically invariant representations.
• Figure 3: Transferability to different text templates. We compare SAFT-L and TGA-ZSR across 13 text templates, using Tiny-ImageNet as the source dataset and PGD-100 as the evaluation method. We report the standard deviations and averaged zero-shot robustness (%) on CIFAR-10, CIFAR-100, and STL-10 for three runs. SAFT-L consistently outperforms TGA-ZSR in averaged zero-shot robust accuracy across all templates. We provide full experimental results in Appendix \ref{['A: text templates']}.