Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness Reprogramming for Representation Learning

Zhichao Hou, MohamadAli Torkamani, Hamid Krim, Xiaorui Liu

TL;DR

The core feature transformation mechanism in representation learning is revisited and a novel non-linear robust pattern matching technique is proposed as a robust alternative to enhance its robustness against adversarial or noisy input perturbations without altering its parameters.

Abstract

This work tackles an intriguing and fundamental open challenge in representation learning: Given a well-trained deep learning model, can it be reprogrammed to enhance its robustness against adversarial or noisy input perturbations without altering its parameters? To explore this, we revisit the core feature transformation mechanism in representation learning and propose a novel non-linear robust pattern matching technique as a robust alternative. Furthermore, we introduce three model reprogramming paradigms to offer flexible control of robustness under different efficiency requirements. Comprehensive experiments and ablation studies across diverse learning models ranging from basic linear model and MLPs to shallow and modern deep ConvNets demonstrate the effectiveness of our approaches. This work not only opens a promising and orthogonal direction for improving adversarial defenses in deep learning beyond existing methods but also provides new insights into designing more resilient AI systems with robust statistics.

Robustness Reprogramming for Representation Learning

TL;DR

The core feature transformation mechanism in representation learning is revisited and a novel non-linear robust pattern matching technique is proposed as a robust alternative to enhance its robustness against adversarial or noisy input perturbations without altering its parameters.

Abstract

This work tackles an intriguing and fundamental open challenge in representation learning: Given a well-trained deep learning model, can it be reprogrammed to enhance its robustness against adversarial or noisy input perturbations without altering its parameters? To explore this, we revisit the core feature transformation mechanism in representation learning and propose a novel non-linear robust pattern matching technique as a robust alternative. Furthermore, we introduce three model reprogramming paradigms to offer flexible control of robustness under different efficiency requirements. Comprehensive experiments and ablation studies across diverse learning models ranging from basic linear model and MLPs to shallow and modern deep ConvNets demonstrate the effectiveness of our approaches. This work not only opens a promising and orthogonal direction for improving adversarial defenses in deep learning beyond existing methods but also provides new insights into designing more resilient AI systems with robust statistics.
Paper Structure (30 sections, 2 theorems, 22 equations, 13 figures, 12 tables, 2 algorithms)

This paper contains 30 sections, 2 theorems, 22 equations, 13 figures, 12 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Robust Nonlinear Pattern Matching
  4. A New Perspective of Representation Learning
  5. Algorithm Development and Analysis
  6. Theoretical Robustness Analysis
  7. Robustness Reprogramming
  8. Experiment
  9. Experimental Setting
  10. Robustness Reprogramming on MLPs
  11. Robustness Reprogramming on Convolution
  12. Robustness Reprogramming on ResNets
  13. Conclusion
  14. Efficient Implementation of Robust Convolution/MLPs
  15. Robust MLPs
  16. ...and 15 more sections

Key Result

Lemma 3.1

Let ${\mathcal{L}}(z)$ be defined in Eq. (eq:robust_estimation), and for any fixed point $z_0$, ${\mathcal{U}}(z,z_0)$ is defined as where $w_d=\frac{1}{2|a_dx_d-z_0/D|}.$ Then, for any $z$, the following holds:

Figures (13)

  • Figure 1: Vanilla Linear Pattern Matching (LPM) vs. Nonlinear Robust Pattern Matching (NRPM).
  • Figure 2: Three Robustness Reprogramming Paradigms : (1) Paradigm 1 freezes the model parameters and treats $\{\lambda\}$ as fixed hyperparameter; (2) Paradigm 2 freezes the model parameters but allows $\{\lambda\}$ to be learnable; (3) Paradigm 3 enables both the model parameters and $\{\lambda\}$ to be learnable.
  • Figure 3: Robustness reprogramming on LeNet. The depth of color represents the size of budget.
  • Figure 4: Adversarial fine-tuning on LeNet.
  • Figure 5: Visualization of hidden embeddings. The LPM-LeNet is more sensitive to perturbation compared to the NRPM-LeNet: (1) When comparing ${\bm{z}}_i$ and ${\bm{z}}_i^\prime$, LPM shows a more significant difference than NRPM. (2) When comparing the likelihood of predictions, the perturbation misleads LPM from predicting 4 to 8, while NRPM consistently predicts 4 in both clean and noisy scenarios.
  • ...and 8 more figures

Theorems & Definitions (6)

  • Lemma 3.1
  • proof
  • Theorem 3.2: Robustness Analysis via Influence Function
  • proof
  • proof
  • proof