Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anderson-type acceleration method for Deep Neural Network optimization

Kazufumi Ito, Tiancheng Xue

TL;DR

The paper tackles slow convergence in neural network optimization by introducing an Anderson-type acceleration (AA) framework for stochastic gradient-based training. It builds theoretical tooling via a reduced-order method (ROM) and residual minimization to accelerate fixed-point-like iterations and compares AA with ROM. The authors demonstrate AA’s applicability to both DNNs and CNNs, applying it to MNIST and observing faster convergence and higher accuracy with fewer epochs, along with data compression considerations in CNNs. The work provides practical algorithmic recipes for integrating AA into NN training and highlights its potential to speed up large-scale neural network optimization. Overall, AA offers a cost-efficient enhancement to inner optimization loops, improving training efficiency without compromising final performance.

Abstract

In this paper we consider the neural network optimization. We develop Anderson-type acceleration method for the stochastic gradient decent method and it improves the network permanence very much. We demonstrate the applicability of the method for Deep Neural Network (DNN) and Convolution Neural Network (CNN).

Anderson-type acceleration method for Deep Neural Network optimization

TL;DR

The paper tackles slow convergence in neural network optimization by introducing an Anderson-type acceleration (AA) framework for stochastic gradient-based training. It builds theoretical tooling via a reduced-order method (ROM) and residual minimization to accelerate fixed-point-like iterations and compares AA with ROM. The authors demonstrate AA’s applicability to both DNNs and CNNs, applying it to MNIST and observing faster convergence and higher accuracy with fewer epochs, along with data compression considerations in CNNs. The work provides practical algorithmic recipes for integrating AA into NN training and highlights its potential to speed up large-scale neural network optimization. Overall, AA offers a cost-efficient enhancement to inner optimization loops, improving training efficiency without compromising final performance.

Abstract

In this paper we consider the neural network optimization. We develop Anderson-type acceleration method for the stochastic gradient decent method and it improves the network permanence very much. We demonstrate the applicability of the method for Deep Neural Network (DNN) and Convolution Neural Network (CNN).
Paper Structure (23 sections, 1 theorem, 40 equations, 3 tables)

This paper contains 23 sections, 1 theorem, 40 equations, 3 tables.

Table of Contents

  1. Introduction
  2. ROM, Anderson type acceleration
  3. Anderson type acceleration method
  4. Reduced order method (ROM)
  5. Comparison of Anderson and ROM
  6. Data compression and CNN
  7. Convolution CNN, data compression
  8. Maxpooling
  9. Batch normalization
  10. Stochastic gradient descent (SGD) for NN design
  11. Stochastic gradient method
  12. Mini-batch method and Anderson-type method
  13. Backpropagation
  14. Convergence Stochastic gradient decent and Kaczmarz method
  15. Description of algorithm
  16. ...and 8 more sections

Key Result

Theorem 2.3

Let $F: X\to X$, consider 2 problems: the first one is And, i.e., with $r_k = \psi(x_k) - y$, we have: and ROM with the constraint $\sum_{k\le m}\alpha_k=1$. Then, for some function $H: X\to X$ such that $\lim_{x_k \to \bar{x}} H(x_k) = 0$ for every $\bar{x}$ close to $x_k$, we have:

Theorems & Definitions (6)

  • Definition 2.1: Anderson type method
  • Definition 2.2: Reduced order method
  • Theorem 2.3
  • Remark 7.1
  • Remark 8.1
  • Remark 8.2