How to Enhance Downstream Adversarial Robustness (almost) without Touching the Pre-Trained Foundation Model?
Meiqi Liu, Zhuoqun Huang, Yue Xing
TL;DR
The paper tackles the high computational cost of adversarial training for foundation models by proposing CRoPD, a robust auto-encoder that preprocesses downstream data without accessing foundation-model weights during training. It formalizes a bound showing the downstream adversarial loss can be controlled by the downstream clean loss plus a scaled adversarial contrastive loss, and it introduces a practical algorithm that optimizes reconstruction loss jointly with adversarial contrastive loss. Extensive experiments across CIFAR-2/10/100, SVHN, ImagenetTe, and Tiny-Imagenet demonstrate that CRoPD yields substantial improvements in downstream robustness under PGD and AutoAttack with only modest degradation in clean accuracy and far lower computational cost than full robust fine-tuning. This work provides a cost-efficient path to robust downstream performance and suggests that feature robustness learned via adversarial contrastive learning is a key driver of downstream resilience, with potential extensions to NLP settings.
Abstract
With the rise of powerful foundation models, a pre-training-fine-tuning paradigm becomes increasingly popular these days: A foundation model is pre-trained using a huge amount of data from various sources, and then the downstream users only need to fine-tune and adapt it to specific downstream tasks. However, due to the high computation complexity of adversarial training, it is not feasible to fine-tune the foundation model to improve its robustness on the downstream task. Observing the above challenge, we want to improve the downstream robustness without updating/accessing the weights in the foundation model. Inspired from existing literature in robustness inheritance (Kim et al., 2020), through theoretical investigation, we identify a close relationship between robust contrastive learning with the adversarial robustness of supervised learning. To further validate and utilize this theoretical insight, we design a simple-yet-effective robust auto-encoder as a data pre-processing method before feeding the data into the foundation model. The proposed approach has zero access to the foundation model when training the robust auto-encoder. Extensive experiments demonstrate the effectiveness of the proposed method in improving the robustness of downstream tasks, verifying the connection between the feature robustness (implied by small adversarial contrastive loss) and the robustness of the downstream task.