CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification

TL;DR

This work tackles robust zero-shot image classification under adversarial perturbations by purifying CLIP embeddings in its latent space rather than in pixel space. It formulates purification risk using bidirectional stochastic differential equations and KL divergence, then introduces two CLIPure variants: CLIPure-Diff (diffusion-prior based likelihood) and CLIPure-Cos (cosine-based likelihood) that operate in CLIP's multi-modal latent space. CLIPure demonstrates state-of-the-art robustness across CIFAR-10, ImageNet, and 13 zero-shot datasets, with CLIPure-Cos achieving substantial efficiency gains by eschewing generative models. The approach requires no dataset-specific training and benefits from CLIP's aligned image-text representations, highlighting the potential of latent, multi-modal purification for broad adversarial robustness applications.

Abstract

In this paper, we aim to build an adversarially robust zero-shot image classifier. We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts ``a photo of a <class-name>.''. Purification is the path we choose since it does not require adversarial training on specific attack types and thus can cope with any foreseen attacks. We then formulate purification risk as the KL divergence between the joint distributions of the purification process of denoising the adversarial samples and the attack process of adding perturbations to benign samples, through bidirectional Stochastic Differential Equations (SDEs). The final derived results inspire us to explore purification in the multi-modal latent space of CLIP. We propose two variants for our CLIPure approach: CLIPure-Diff which models the likelihood of images' latent vectors with the DiffusionPrior module in DaLLE-2 (modeling the generation process of CLIP's latent vectors), and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and ``a photo of a.''. As far as we know, CLIPure is the first purification method in multi-modal latent space and CLIPure-Cos is the first purification method that is not based on generative models, which substantially improves defense efficiency. We conducted extensive experiments on CIFAR-10, ImageNet, and 13 datasets that previous CLIP-based defense methods used for evaluating zero-shot classification robustness. Results show that CLIPure boosts the SOTA robustness by a large margin, e.g., from 71.7% to 91.1% on CIFAR10, from 59.6% to 72.6% on ImageNet, and 108% relative improvements of average robustness on the 13 datasets over previous SOTA. The code is available at https://github.com/TMLResearchGroup-CAS/CLIPure.

Figures (13)

• Figure 1: Adversarial robustness of two CLIPure versions versus adversarially trained CLIP models, evaluated against AutoAttack with $\ell_{\infty}=4/255$ across 13 zero-shot classification datasets.
• Figure 2: Negative log-likelihood estimated by diffusion models on (a) pixel space via EDM, (b) uni-modal latent space via VQVAE, (c) multi-modal latent space via DiffusionPrior, and (d) multi-modal latent space via CLIP (using cosine similarity for log-likelihood estimation). KL represents the value of $\mathrm{KL}(p(\boldsymbol{x}_{\text{adv}}) \| p(\boldsymbol{x}_{\text{ben}}))$ discussed in Section \ref{['section: purification in CLIP latent space']} indicating the difference between clean and adversarial example distribution.
• Figure 3: Accuracy and robustness (detailed in Table \ref{['table: performance across backbones']}) of CLIPure-Cos for (Left) various versions of CLIP and (Right) different backbone models. The bubble size represents the number of parameters, which is also indicated alongside each bubble. The left figure presents CLIPure-Cos based on ResNet-based models (including RN50, RN101, RN50x64, marked in red) and ViT-based models (ViT-B-16, ViT-B-32, ViT-L-14, ViT-H-14, and ViT-bigG-14, marked in blue). The right figure depicts CLIPure-Cos based on CLIP (including ViT-B-16, ViT-L-14, ViT-H-14, ViT-bigG-14, marked in blue), EVA2-CLIP (including ViT-B-16 and ViT-L-14, marked in green), CLIPA (including ViT-L-14 and ViT-H-14, marked in red), and CoCa (ViT-B-32, marked in yellow). The blue dashed line represents the point where robust accuracy equals clean accuracy, serving as the upper bound of robustness, since successful defense against adversarial attacks hinges on accurate classification.
• Figure 4: Comparison of the Schrödinger Bridge schrodinger1932theorie framework with our adversarial attack and purification process
• Figure 5: Illustration of the process of CLIPure including the purification in latent space and zero-shot classification, detailed in Algorithm \ref{['algorithm: latent purification by CLIP and DaLLE2.DiffusionPrior']}.