Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Theoretical View of Linear Backpropagation and Its Convergence

Ziang Li, Yiwen Guo, Haodi Liu, Changshui Zhang

TL;DR

It is demonstrated that, somewhat surprisingly, LinBP can lead to faster convergence in neural-network-involved learning tasks, including adversarial attack and model training, in the same hyper-parameter settings, compared to BP.

Abstract

Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by Guo et al. Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al.'s paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments.

A Theoretical View of Linear Backpropagation and Its Convergence

TL;DR

It is demonstrated that, somewhat surprisingly, LinBP can lead to faster convergence in neural-network-involved learning tasks, including adversarial attack and model training, in the same hyper-parameter settings, compared to BP.

Abstract

Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by Guo et al. Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al.'s paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments.
Paper Structure (18 sections, 3 theorems, 26 equations, 12 figures, 2 tables)

This paper contains 18 sections, 3 theorems, 26 equations, 12 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Preliminary Knowledge
  3. Model Training
  4. White-box Adversarial Attack
  5. Linear Backpropagation
  6. Theoretical Analyses
  7. Theoretical Analyses for Adversarial Attack
  8. Theoretical Analyses for Model Training
  9. Discussions about $\eta$ and $t$
  10. Experimental Results
  11. Simulation Experiments
  12. Adversarial attack
  13. Model training
  14. More Practical Experiments
  15. Adversarial attack on DNNs
  16. ...and 3 more sections

Key Result

Lemma 1

$G(\mathbf{e},\mathbf{x}) := \mathbf{W}^TD(\mathbf{W},\mathbf{e})\mathbf{V}^T\mathbf{V}D(\mathbf{W},\mathbf{x})\mathbf{Wx}$, where $\mathbf{e} \in \mathbb{R}^{d_1}$ is a unit vector, $\mathbf{x} \in \mathbb{R}^{d_1}$ is the input data vector, $\mathbf{W}\in\mathbb{R}^{d_2 \times d_1}$ and $\mathbf{V where $\Theta \in [0,\pi]$ is the angle between $\mathbf{e}$ and $\mathbf{x}$.

Figures (12)

  • Figure 1: LinBP leads to more powerful white-box adversarial examples which are closer to the optimal ones.
  • Figure 2: Using normalized gradients, LinBP still leads to more powerful white-box adversarial examples.
  • Figure 3: LinBP leads to lower training loss and better approximation to the teacher model, especially at the early stage of training.
  • Figure 4: LinBP leads to similar training loss and approximation to the teacher model with $l_2$ normalized gradients.
  • Figure 5: Compare the training loss and training accuracy of the MLP and LeNet-5 on MNIST, using LinBP or BP.
  • ...and 7 more figures

Theorems & Definitions (3)

  • Lemma 1
  • Theorem 1
  • Theorem 2