Proxy Robustness in Vision Language Models is Effortlessly Transferable

TL;DR

Vision-Language Models like CLIP exhibit proxy adversarial robustness across heterogeneous architectures, enabling robustness transfer without training robust teachers. The authors propose Heterogeneous Proxy Transfer with Generalization-Pivot Decoupling (HPT-GPD), a two-stage training scheme—generalization-anchored warm-up followed by generalization-pulled HPT with EMA—to simultaneously boost zero-shot adversarial robustness and preserve natural generalization. They formalize the approach with loss terms $L_{ ext{GA}}$ and $L_{ ext{RT-CLIP}}$, provide theoretical bounds on the target risk, and validate on 15 datasets, showing robustness gains over state-of-the-art baselines like PMG and FT-TeCoA under both PGD and AutoAttack. The results suggest a practical path for robust, scalable VLM deployment and offer insights into proxy-based defense mechanisms, with broader implications for cross-architecture robustness in multi-modal learning.

Abstract

As a pivotal technique for improving the defense of deep models, adversarial robustness transfer via distillation has demonstrated remarkable success in conventional image classification tasks. However, this paradigm encounters critical challenges when applied to vision-language models (VLM) (e.g., CLIP): constructing adversarially robust teacher for large-scale multi-modal models demands prohibitively high computational resources. We bridge this gap by revealing an interesting phenomenon: vanilla CLIP (without adversarial training) exhibits intrinsic defensive capabilities against adversarial examples generated by another CLIP with different architectures. We formally define this as proxy adversarial robustness, and naturally propose a Heterogeneous Proxy Transfer (HPT) framework that establishes cross-architectural robustness distillation channels between CLIP variants, effortlessly enabling the VLM robustness transfer from proxy to target models. Yet, such proxy transfer paradigm easily induces severe overfitting, leading to a sharp degradation in zero-shot natural generalization. To resolve that, we design Generalization-Pivot Decoupling (GPD) by leveraging the difference in learning rate scheduling. This decouples the proxy transfer process into a generalization-anchored warm-up that maintains generalization and a generalization-pulled HPT that promotes adversarial robustness, to achieve an equilibrium between natural generalization and adversarial robustness. Extensive experiments on 15 zero-shot datasets demonstrate the effectiveness of our HPT-GPD method. The code is available at the website of github.com/fxw13/HPT-GPD.

Figures (13)

• Figure 2: When performing adversarial robustness transfer from ViT-B/16-based CLIP to ViT-B/32-based CLIP, the zero-shot generalization of the target CLIP is significantly impacted, leading to substantial degradation. After deploying our method, the performance on natural samples across downstream datasets is effectively preserved.
• Figure 3: The pipeline of the proposed HPT-GPD. In the generalization-anchored warm-up, the zero-shot generalization ability is maintained at a low learning rate; then, the proxy robust transfer is performed at a high learning rate, while the model obtained in the warm-up is used for generalization pulling.
• Figure 4: When using ViT-B/16-based CLIP as the proxy and ViT-B/32-based CLIP as the target model, the natural performance is reduced after the robust transfer via Eq. \ref{['eq4']}.
• Figure 5: Training loss (left) and relative $L_2$ distance of parameters between fine-tuned CLIP and original ViT-B/32-based CLIP (right) for AFT at different learning rates on the TinyImageNet dataset.
• Figure 6: The influence of learning rate. The figures on the left and right represent the learning rates in the Generalization-Pulled HPT and Generalization-Anchored Warm-up stages, respectively.

Theorems & Definitions (2)

• theorem 1
• theorem 2