Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models

Kangtao Lv, Huangsen Cao, Kainan Tu, Yihuai Xu, Zhimeng Zhang, Xin Ding, Yongwei Wang

TL;DR

Hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously, is proposed, which significantly enhances the adversarial robustness of pretrained large vision models without excessive computational overhead.

Abstract

Large vision models have been found vulnerable to adversarial examples, emphasizing the need for enhancing their adversarial robustness. While adversarial training is an effective defense for deep convolutional models, it often faces scalability issues with large vision models due to high computational costs. Recent approaches propose robust fine-tuning methods, such as adversarial tuning of low-rank adaptation (LoRA) in large vision models, but they still struggle to match the accuracy of full parameter adversarial fine-tuning. The integration of various defense mechanisms offers a promising approach to enhancing the robustness of large vision models, yet this paradigm remains underexplored. To address this, we propose hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously. Specifically, adversarial tuning of each defense method is formulated as a learning task, and a hypernetwork generates LoRA specific to this defense. Then, a random sampling and tuning strategy is proposed to extract and facilitate the defensive knowledge transfer between different defenses. Finally, diverse LoRAs are merged to enhance the adversarial robustness. Experiments on various datasets and model architectures demonstrate that HyperAT significantly enhances the adversarial robustness of pretrained large vision models without excessive computational overhead, establishing a new state-of-the-art benchmark.

Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models

TL;DR

Hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously, is proposed, which significantly enhances the adversarial robustness of pretrained large vision models without excessive computational overhead.

Abstract

Large vision models have been found vulnerable to adversarial examples, emphasizing the need for enhancing their adversarial robustness. While adversarial training is an effective defense for deep convolutional models, it often faces scalability issues with large vision models due to high computational costs. Recent approaches propose robust fine-tuning methods, such as adversarial tuning of low-rank adaptation (LoRA) in large vision models, but they still struggle to match the accuracy of full parameter adversarial fine-tuning. The integration of various defense mechanisms offers a promising approach to enhancing the robustness of large vision models, yet this paradigm remains underexplored. To address this, we propose hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously. Specifically, adversarial tuning of each defense method is formulated as a learning task, and a hypernetwork generates LoRA specific to this defense. Then, a random sampling and tuning strategy is proposed to extract and facilitate the defensive knowledge transfer between different defenses. Finally, diverse LoRAs are merged to enhance the adversarial robustness. Experiments on various datasets and model architectures demonstrate that HyperAT significantly enhances the adversarial robustness of pretrained large vision models without excessive computational overhead, establishing a new state-of-the-art benchmark.
Paper Structure (16 sections, 7 equations, 3 figures, 7 tables, 2 algorithms)

This paper contains 16 sections, 7 equations, 3 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminary
  5. HyperAT
  6. HyperAT+
  7. Experiments
  8. Experimental Setup
  9. Results and Analysis
  10. Parameter Efficiency Analysis
  11. Ablation Studies
  12. Conclusion
  13. Appendix A
  14. Appendix B
  15. Appendix C
  16. ...and 1 more sections

Figures (3)

  • Figure 1: The HyperAT framework involves two stages: a) generating weights for a mixture of defensive LoRAs, and b) merging weights to capture more sophisticated decision boundaries.
  • Figure 2: Illustration of the proposed HyperAT architecture. The ViT model integrates HyperAT within both the attention and feed-forward blocks. HyperAT comprises three main components: 1) learned method embedding generator, 2) shared Hypernetwork, and 3) method-specific LoRA modules. We employ an embedding generator to produce method-specific embeddings for each adversarial training method. The shared hypernetwork then takes embeddings as input to generate the parameters for the method-specific LoRA module, which are used for adversarial fine-tuning with a small number of trainable parameters.
  • Figure 3: The effect of different iterations for HyperAT+