Initialization Matters for Adversarial Transfer Learning

TL;DR

The paper addresses adversarial robustness in transfer learning under the Pretraining-Finetuning paradigm and shows that initialization, particularly a robust pretrained backbone and a robustly initialized linear head, is critical. It systematically compares six finetuning methods, finds that adversarial robustness deteriorates when starting from nonrobust pretraining, and introduces Robust Linear Initialization (RoLI), which initializes the linear head with weights from adversarial linear probing to maximize inherited robustness and domain adaptation. RoLI, when combined with adversarial finetuning, delivers state-of-the-art robustness across five image classification datasets, with an average clean accuracy gain of 3.88% and robust accuracy gain of 2.44% over random initialization. The work also analyzes the relationship between transferred robustness and transferred accuracy, demonstrates that RoLI mitigates overfitting, and provides practical training details and code to facilitate adoption in adversarial transfer learning.

Abstract

With the prevalence of the Pretraining-Finetuning paradigm in transfer learning, the robustness of downstream tasks has become a critical concern. In this work, we delve into adversarial robustness in transfer learning and reveal the critical role of initialization, including both the pretrained model and the linear head. First, we discover the necessity of an adversarially robust pretrained model. Specifically, we reveal that with a standard pretrained model, Parameter-Efficient Finetuning (PEFT) methods either fail to be adversarially robust or continue to exhibit significantly degraded adversarial robustness on downstream tasks, even with adversarial training during finetuning. Leveraging a robust pretrained model, surprisingly, we observe that a simple linear probing can outperform full finetuning and other PEFT methods with random initialization on certain datasets. We further identify that linear probing excels in preserving robustness from the robust pretraining. Based on this, we propose Robust Linear Initialization (RoLI) for adversarial finetuning, which initializes the linear head with the weights obtained by adversarial linear probing to maximally inherit the robustness from pretraining. Across five different image classification datasets, we demonstrate the effectiveness of RoLI and achieve new state-of-the-art results. Our code is available at \url{https://github.com/DongXzz/RoLI}.

Figures (8)

• Figure 1: Robust Linear Initialization (RoLI), significantly improves adversarial robustness. RoLI, denoted in red, achieves an average 3.88% increase in clean accuracy and a 2.44% increase in robust accuracy compared to Random Linear Initialization (RanLI) of the linear head during adversarial finetuning, averaged across five downstream datasets. Our best-performing RoLI - Full-FT, which represents adversarial full finetuning with robust linear initialization, achieves a new state-of-the-art performance. We include six popular finetuning methods with Swin Transformer swin and two existing state-of-the-art techniques for adversarial transfer learning: TWINS-AT twins and AutoLoRA autolora.
• Figure 2: Illustration of six different finetuning techniques, arranged in descending order according to the number of tunable parameters.
• Figure 3: PEFT methods fail or exhibit significantly degraded performance with a standard pretrained model (in blue and gray). Models finetuned from a robust pretrained model (in red) exhibit high accuracy and robustness, as they are consistently positioned in the top-right corner. Among six finetuning techniques, Full-FT outperforms others when starting with a standard pretrained model; however, it falls short of other methods when starting with a robust pretrained model. The specific numerical results are provided in the supplementary.
• Figure 4: Linear probing with adversarial finetuning surpasses other methods on Caltech256. With standard finetuning, the PEFT methods exhibit inherited robustness (Robustness $>$ 0) from the pretraining. Notably, linear probing surpasses the other methods by a substantial margin in robustness while maintaining similar clean data accuracy on Caltech256. Additionally, adversarial training during finetuning is effective in enhancing all methods' robustness.
• Figure 5: Transferred robustness of adversarial linear probing strongly correlates with transferred accuracy of standard linear probing.