Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing

Chengyan Fu, Wenjie Wang

TL;DR

PEFTSmoothing addresses the practical bottleneck of obtaining certifiable robustness under $l_2$ perturbations without retraining large models from scratch. By leveraging Parameter-Efficient Fine-Tuning methods (Prompt-tuning, LoRA, Adapter) to learn Gaussian-noise-augmented data, it produces a PEFTSmoothed classifier compatible with randomized smoothing guarantees. Empirical results on CIFAR-10 and ImageNet show superior certified accuracy and dramatically reduced training parameters versus denoising-based approaches, with a viable black-box variant and integration with downstream PEFT for task adaptation. The work demonstrates a scalable, efficient pathway to robust deployment of large vision models under adversarial perturbations.

Abstract

Randomized smoothing is the primary certified robustness method for accessing the robustness of deep learning models to adversarial perturbations in the l2-norm, by adding isotropic Gaussian noise to the input image and returning the majority votes over the base classifier. Theoretically, it provides a certified norm bound, ensuring predictions of adversarial examples are stable within this bound. A notable constraint limiting widespread adoption is the necessity to retrain base models entirely from scratch to attain a robust version. This is because the base model fails to learn the noise-augmented data distribution to give an accurate vote. One intuitive way to overcome this challenge is to involve a custom-trained denoiser to eliminate the noise. However, this approach is inefficient and sub-optimal. Inspired by recent large model training procedures, we explore an alternative way named PEFTSmoothing to adapt the base model to learn the Gaussian noise-augmented data with Parameter-Efficient Fine-Tuning (PEFT) methods in both white-box and black-box settings. Extensive results demonstrate the effectiveness and efficiency of PEFTSmoothing, which allow us to certify over 98% accuracy for ViT on CIFAR-10, 20% higher than SoTA denoised smoothing, and over 61% accuracy on ImageNet which is 30% higher than CNN-based denoiser and comparable to the Diffusion-based denoiser.

Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing

TL;DR

PEFTSmoothing addresses the practical bottleneck of obtaining certifiable robustness under perturbations without retraining large models from scratch. By leveraging Parameter-Efficient Fine-Tuning methods (Prompt-tuning, LoRA, Adapter) to learn Gaussian-noise-augmented data, it produces a PEFTSmoothed classifier compatible with randomized smoothing guarantees. Empirical results on CIFAR-10 and ImageNet show superior certified accuracy and dramatically reduced training parameters versus denoising-based approaches, with a viable black-box variant and integration with downstream PEFT for task adaptation. The work demonstrates a scalable, efficient pathway to robust deployment of large vision models under adversarial perturbations.

Abstract

Randomized smoothing is the primary certified robustness method for accessing the robustness of deep learning models to adversarial perturbations in the l2-norm, by adding isotropic Gaussian noise to the input image and returning the majority votes over the base classifier. Theoretically, it provides a certified norm bound, ensuring predictions of adversarial examples are stable within this bound. A notable constraint limiting widespread adoption is the necessity to retrain base models entirely from scratch to attain a robust version. This is because the base model fails to learn the noise-augmented data distribution to give an accurate vote. One intuitive way to overcome this challenge is to involve a custom-trained denoiser to eliminate the noise. However, this approach is inefficient and sub-optimal. Inspired by recent large model training procedures, we explore an alternative way named PEFTSmoothing to adapt the base model to learn the Gaussian noise-augmented data with Parameter-Efficient Fine-Tuning (PEFT) methods in both white-box and black-box settings. Extensive results demonstrate the effectiveness and efficiency of PEFTSmoothing, which allow us to certify over 98% accuracy for ViT on CIFAR-10, 20% higher than SoTA denoised smoothing, and over 61% accuracy on ImageNet which is 30% higher than CNN-based denoiser and comparable to the Diffusion-based denoiser.
Paper Structure (14 sections, 1 theorem, 8 equations, 8 figures, 2 tables)

This paper contains 14 sections, 1 theorem, 8 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Randomized Smoothing
  4. Denoised Smoothing
  5. Intuition: PEFT Guides Large Vision Models to Learn Noise-Augmented Data Distribution
  6. PEFTSmoothing
  7. Experiments
  8. Configuration
  9. Certified Accuracy of PEFTSmoothing
  10. CIFAR-10 Ablation Studies
  11. Black-box PEFTSmoothing
  12. Two Birds One Stone
  13. Related Work
  14. Conclusion

Key Result

Theorem 2.1

Given a deterministic classifier $\mathcal{F}$ and its probabilistic counterpart $\mathcal{G}$ defined in Equation equ:g, let $\delta\sim\mathcal{N}(0,\sigma^2I)$, suppose $c_A$ is the most probable class, and $\:\underline{p_A},\overline{p_B}\in[0,1]\:$ satisfy: Then $\mathcal{G}(x+\delta)=c_A$ for all $\:\|\delta\|_2<$R$,\:$ where

Figures (8)

  • Figure 1: Illustration of PEFTSmoothing procedures. We incorporate four PEFT methods to transform a base classifier PEFTSmoothed Classifier. The final output is the majority votes of The noised-augmented inputs, which is stable within a certain norm bound.
  • Figure 2: Comparing the noise-augmented data learning capacity of Prompt-tuning and denoiser
  • Figure 3: Certified accuracy comparison on PEFTSmoothing and denoised smoothing
  • Figure 4: Test Accuracy vs. Size of Trained Parameters
  • Figure 5: Certified Accuracy of Prompt-tuning in PEFTSmoothing with Different Prompt Lengths. $\sigma=0.5$
  • ...and 3 more figures

Theorems & Definitions (1)

  • Theorem 2.1