Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness

Suraj Srinivas, Sebastian Bordt, Hima Lakkaraju

TL;DR

This work provides a first explanation of PAGs via off-manifold robustness, which states that models must be more robust off- the data manifold than they are on-manifold, and demonstrates theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment.

Abstract

One of the remarkable properties of robust computer vision models is that their input-gradients are often aligned with human perception, referred to in the literature as perceptually-aligned gradients (PAGs). Despite only being trained for classification, PAGs cause robust models to have rudimentary generative capabilities, including image generation, denoising, and in-painting. However, the underlying mechanisms behind these phenomena remain unknown. In this work, we provide a first explanation of PAGs via \emph{off-manifold robustness}, which states that models must be more robust off- the data manifold than they are on-manifold. We first demonstrate theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment. We then show that Bayes optimal models satisfy off-manifold robustness, and confirm the same empirically for robust models trained via gradient norm regularization, randomized smoothing, and adversarial training with projected gradient descent. Quantifying the perceptual alignment of model gradients via their similarity with the gradients of generative models, we show that off-manifold robustness correlates well with perceptual alignment. Finally, based on the levels of on- and off-manifold robustness, we identify three different regimes of robustness that affect both perceptual alignment and model accuracy: weak robustness, bayes-aligned robustness, and excessive robustness. Code is available at \url{https://github.com/tml-tuebingen/pags}.

Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness

TL;DR

This work provides a first explanation of PAGs via off-manifold robustness, which states that models must be more robust off- the data manifold than they are on-manifold, and demonstrates theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment.

Abstract

One of the remarkable properties of robust computer vision models is that their input-gradients are often aligned with human perception, referred to in the literature as perceptually-aligned gradients (PAGs). Despite only being trained for classification, PAGs cause robust models to have rudimentary generative capabilities, including image generation, denoising, and in-painting. However, the underlying mechanisms behind these phenomena remain unknown. In this work, we provide a first explanation of PAGs via \emph{off-manifold robustness}, which states that models must be more robust off- the data manifold than they are on-manifold. We first demonstrate theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment. We then show that Bayes optimal models satisfy off-manifold robustness, and confirm the same empirically for robust models trained via gradient norm regularization, randomized smoothing, and adversarial training with projected gradient descent. Quantifying the perceptual alignment of model gradients via their similarity with the gradients of generative models, we show that off-manifold robustness correlates well with perceptual alignment. Finally, based on the levels of on- and off-manifold robustness, we identify three different regimes of robustness that affect both perceptual alignment and model accuracy: weak robustness, bayes-aligned robustness, and excessive robustness. Code is available at \url{https://github.com/tml-tuebingen/pags}.
Paper Structure (31 sections, 4 theorems, 7 equations, 17 figures)

This paper contains 31 sections, 4 theorems, 7 equations, 17 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robust training of neural networks
  4. Gradient-based model explanations
  5. Explaining Perceptually-Aligned Gradients
  6. Notation
  7. Off-Manifold Robustness $\Leftrightarrow$ On-Manifold Gradient Alignment
  8. Connecting Bayes Optimal Classifiers and Off-Manifold Robustness
  9. Bayes optimal classifiers.
  10. Distinguishing Between the Data and the Signal Manifold.
  11. Connecting Robust Models and Off-Manifold Robustness
  12. Argument 1: Combined robust objectives are non-isotropic.
  13. Argument 2: Robust linear models are off-manifold robust.
  14. Experimental Evaluation
  15. Evaluating On- vs. Off-manifold Robustness of Models
  16. ...and 16 more sections

Key Result

Theorem 3.1

A function $f:\mathbb{R}^d\to\mathbb{R}$ exhibits on-manifold gradient alignment if and only if it is off-manifold robust wrt normal noise $\mathbf{u} \sim \mathcal{N}(0, \sigma^2)$ for $\sigma \rightarrow 0$ (with $\rho_1 = \rho_2$).

Figures (17)

  • Figure 1: A demonstration of the perceptual alignment phenomenon. The input-gradients of robust classifiers ("robust gradient") are perceptually similar to the score of diffusion models karras2022elucidating, while being qualitatively distinct from input-gradients of standard models ("standard gradient"). Best viewed in digital format.
  • Figure 2: An illustration of a signal-distractor decomposition for a bird classification task. The signal represents the discriminative parts of the input, while the distractor represents the non-discriminative parts.
  • Figure 3: Top Row: Robust models are off-manifold robust. The figure depicts the inverse on- and off-manifold robustness of Resnet18 models trained with different objectives on CIFAR-10 (larger values correspond to less robustness). As we increase the importance of the robustness term in the training objective, the models become increasingly robust to off-manifold perturbations. At the same time, their robustness to on-manifold perturbations stays approximately constant. This means that the models become off-manifold robust. As we further increase the degree of robustness, both on- and off-manifold robustness increase. Bottom Row: The input gradients of robust models are perceptually similar to the score of the probability distribution, as measured by the LPIPS metric. We can also identify the models that have the most perceptually-aligned gradients (the global maxima of the yellow curves). Figures depict mean and minimum/maximum across 10 different random seeds. Additional results including the smoothness penalty can be found in Supplement Figure \ref{['fig:apx_perturbation_size']}.
  • Figure 4: The input gradients of robust models trained with projected gradient descent on ImageNet and Imagenet-64x64 are perceptually similar to the score of the probability distribution, as measured by the LPIPS metric. On Imagenet-64x64, we also trained excessively robust models.
  • Figure 5: Robust Models are relatively robust to noise on a distractor.
  • ...and 12 more figures

Theorems & Definitions (9)

  • Definition 1
  • Definition 2
  • Theorem 3.1: Equivalence between off-manifold robustness and on-manifold alignment
  • Definition 3
  • Theorem 3.2
  • Theorem A.1: Equivalence between off-manifold robustness and on-manifold alignment
  • proof
  • Theorem A.2
  • proof