Adversarial Robustification via Text-to-Image Diffusion Models

TL;DR

This work tackles the challenge of achieving adversarial robustness for off-the-shelf vision models without access to training data. It introduces a scalable, model-agnostic framework that uses text-to-image diffusion models as zero-shot denoisers within a denoise-and-classify pipeline to obtain provable robustness, complemented by two self-adaptation schemes that synthesize reference images and personalize the diffusion model while regularizing the classifier. The approach yields strong data-free robustness improvements on CLIP and a ResNet-50 across diverse datasets, often surpassing data-dependent baselines and achieving competitive certified robustness. The results demonstrate practical robustness gains for real-world, data-constrained deployments and open questions for extending data-free defenses to broader model classes and black-box settings.

Abstract

Adversarial robustness has been conventionally believed as a challenging property to encode for neural networks, requiring plenty of training data. In the recent paradigm of adopting off-the-shelf models, however, access to their training data is often infeasible or not practical, while most of such models are not originally trained concerning adversarial robustness. In this paper, we develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data. Our intuition is to view recent text-to-image diffusion models as "adaptable" denoisers that can be optimized to specify target tasks. Based on this, we propose: (a) to initiate a denoise-and-classify pipeline that offers provable guarantees against adversarial attacks, and (b) to leverage a few synthetic reference images generated from the text-to-image model that enables novel adaptation schemes. Our experiments show that our data-free scheme applied to the pre-trained CLIP could improve the (provable) adversarial robustness of its diverse zero-shot classification derivatives (while maintaining their accuracy), significantly surpassing prior approaches that utilize the full training data. Not only for CLIP, we also demonstrate that our framework is easily applicable for robustifying other visual classifiers efficiently.

Figures (9)

• Figure 1: Comparison of clean and robust ($\|\varepsilon\|_2 \le 1.0$) accuracy on zero-shot classification: our framework (a) not only maintains the original accuracy of CLIP radford2021learning; but also (b) significantly improves its robust accuracy, e.g. compared to Mao et al. mao2023understanding.
• Figure 2: An overview of the proposed framework: (a) during inference, we perform denoised smoothing with a self-personalized text-to-image diffusion model, having provable guarantees on adversarial robustness (\ref{['sec:method:denoise-text-image-model']}); (b) by utilizing synthetic references from the text-to-image model, one can adapt both diffusion model and classifier for robustness (\ref{['sec:method:zero-shot-adaptation']}).
• Figure 3: Qualitative comparisons of denoised images on varying correct factor $k$. We compared the denoised outputs from \ref{['eq:one-step-denoising-super']} under Gaussian noise of $\sigma=0.25$.
• Figure 4: ImageNet accuracy (%) on varying correction factor $k$.
• Figure 5: Comparison of the top-5 concepts with the highest similarity to an input image (labeled "Volleyball") before and after an $\ell_2$-adversarial attack at $\varepsilon=1.0$. Unlike other methods, our proposed framework consistently maintains relevant concepts.